% !TeX spellcheck = en_US

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Author template for Management Science (mnsc) for articles with no e-companion (EC)
%% Mirko Janc, Ph.D., INFORMS, mirko.janc@informs.org
%% ver. 0.95, December 2010
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\documentclass[mnsc,blindrev]{informs3}
%\documentclass[mnsc,nonblindrev]{informs3} % current default for manuscript submission

\OneAndAHalfSpacedXI
%%\OneAndAHalfSpacedXII % Current default line spacing
%%\DoubleSpacedXII
%%\DoubleSpacedXI

% If hyperref is used, dvi-to-ps driver of choice must be declared as
%   an additional option to the \documentclass. For example
%\documentclass[dvips,mnsc]{informs3}      % if dvips is used
%\documentclass[dvipsone,mnsc]{informs3}   % if dvipsone is used, etc.

% Private macros here (check that there is no clash with the style)

% Natbib setup for author-year style
\usepackage{natbib}
 \bibpunct[, ]{(}{)}{,}{a}{}{,}%
 \def\bibfont{\small}%
 \def\bibsep{\smallskipamount}%
 \def\bibhang{24pt}%
 \def\newblock{\ }%
 \def\BIBand{and}%

%% Setup of theorem styles. Outcomment only one.
%% Preferred default is the first option.
\TheoremsNumberedThrough     % Preferred (Theorem 1, Lemma 1, Theorem 2)
%\TheoremsNumberedByChapter  % (Theorem 1.1, Lema 1.1, Theorem 1.2)
\ECRepeatTheorems

%% Setup of the equation numbering system. Outcomment only one.
%% Preferred default is the first option.
\EquationsNumberedThrough    % Default: (1), (2), ...
%\EquationsNumberedBySection % (1.1), (1.2), ...

% For new submissions, leave this number blank.
% For revisions, input the manuscript number assigned by the on-line
% system along with a suffix ".Rx" where x is the revision number.
\MANUSCRIPTNO{1} % When the article is logged in and DOI assigned to it,
                 %   this manuscript number is no longer necessary

%%%%%%%%%%%%%%%%



%%%%%%%%%%%%%%%%
%Reza's Preamble
\usepackage{amsmath}
\usepackage{float}
\usepackage[caption=false]{subfig}
\usepackage{multirow} 
\usepackage{booktabs}
\usepackage{extarrows}
\usepackage{color}
\usepackage{bm}
\usepackage{setspace}
%\usepackage[colorlinks=true, citecolor=blue]{hyperref}
\usepackage[subnum]{cases}
\newcommand{\surv}[2]{\mathbf{\overline{F}}_{#1}(#2)}
\newcommand{\pdf}[2]{\mathbf{{f}}_{#1}(#2)}
\newcommand{\hrate}[2] {\mathbf{r}_{#1}(#2)}
\newcommand{\pr}{\mathbb{P}\/}
\newcommand{\Ex}{\mathbf{\mathbb{E\/}}}
\newcommand{\diff}[2]{\frac{d}{d #2}#1}
\newcommand{\diffp}[2]{\frac{\partial}{\partial #2}#1}
\newcommand{\noat}[1]{\textcolor{red}{ #1}}
%%%%%%%%%%%%%%%%


\begin{document}
%%%%%%%%%%%%%%%%temporary


%%%%%%%%%%%%%%%%

% Outcomment only when entries are known. Otherwise leave as is and
%   default values will be used.
%\setcounter{page}{1}
%\VOLUME{00}%
%\NO{0}%
%\MONTH{Xxxxx}% (month or a similar seasonal id)
%\YEAR{0000}% e.g., 2005
%\FIRSTPAGE{000}%
%\LASTPAGE{000}%
%\SHORTYEAR{00}% shortened year (two-digit)
%\ISSUE{0000} %
%\LONGFIRSTPAGE{0001} %
%\DOI{10.1287/xxxx.0000.0000}%

% Author's names for the running heads
% Sample depending on the number of authors;
% \RUNAUTHOR{Jones}
% \RUNAUTHOR{Jones and Wilson}
% \RUNAUTHOR{Jones, Miller, and Wilson}
% \RUNAUTHOR{Jones et al.} % for four or more authors
% Enter authors following the given pattern:
 \RUNAUTHOR{Skandari et al.}

% Title or shortened title suitable for running heads. Sample:
% \RUNTITLE{Bundling Information Goods of Decreasing Value}
% Enter the (shortened) title:
\RUNTITLE{Optimal Vascular Access for HD Patients}

% Full title. Sample:
% \TITLE{Bundling Information Goods of Decreasing Value}
% Enter the full title:
\TITLE{Optimal Vascular Access Choice for Patients on Hemodialysis}

% Block of authors and their affiliations starts here:
% NOTE: Authors with same affiliation, if the order of authors allows,
%   should be entered in ONE field, separated by a comma.
%   \EMAIL field can be repeated if more than one author
\ARTICLEAUTHORS{%

\AUTHOR{M. Reza Skandari} \AFF{Operations and Logistics Division,
Sauder School of Business, University of British Columbia, 2053 Main
Mall, Vancouver, British Columbia V6T 1Z2, Canada,
\EMAIL{reza.skandari@sauder.ubc.ca}} %, \URL{}}

\AUTHOR{Steven M. Shechter} \AFF{Operations and Logistics Division,
Sauder School of Business, University of British Columbia, 2053 Main
Mall, Vancouver, British Columbia V6T 1Z2, Canada,
\EMAIL{steven.shechter@sauder.ubc.ca}} %, \URL{}}

\AUTHOR{Nadia Zalunardo} \AFF{Division of Nephrology,
Department of Medicine, University of British Columbia, 5th Floor, 2775 Laurel St.
Vancouver, British Columbia V5Z 1M9, Canada,
\EMAIL{nadia.zalunardo@vch.ca}} %, \URL{}}

% Enter all authors
} % end of the block

\ABSTRACT{%
Which vascular access to use is considered one of the most important questions in the care of patients on hemodialysis (HD). An arteriovenous fistula (AVF) is often considered the gold standard for delivering HD due to better patient survival, higher quality of life, and fewer complications.
However, AVFs have some limitations as they require surgery, it takes approximately three months to know if the surgery was successful or not, and a majority of these surgeries end in failure. On the other hand, another common vascular access, the central venous catheter (CVC), can be inserted via a simple procedure and used immediately after placement.
In this research, we address the question of whether and when to refer incident and established HD patients for AVF, with the aim to find individualized policies that maximize a patient's probability of survival and remaining quality adjusted life expectancy (QALE). Using a continuous-time dynamic programming model and under certain data-driven assumptions, we establish structural properties of the optimal policy for each objective. We provide further insights for policy makers through our numerical experiments.

% Enter your abstract
}%

% Sample
%\KEYWORDS{deterministic inventory theory; infinite linear programming duality;
%  existence of optimal policies; semi-Markov decision process; cyclic schedule}

% Fill in data. If unknown, outcomment the field
\KEYWORDS{arteriovenous fistula, central venous catheter, hemodialysis, dynamic programming, medical decision making, optimal treatment policies} 
%\HISTORY{This paper was first submitted on \today.}

\maketitle
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% Samples of sectioning (and labeling) in OPRE
% NOTE: (1) \section and \subsection do NOT end with a period
%       (2) \subsubsection and lower need end punctuation
%       (3) capitalization is as shown (title style).
%
%\section{Introduction.}\label{intro} %%1.
%\subsection{Duality and the Classical EOQ Problem.}\label{class-EOQ} %% 1.1.
%\subsection{Outline.}\label{outline1} %% 1.2.
%\subsubsection{Cyclic Schedules for the General Deterministic SMDP.}
%  \label{cyclic-schedules} %% 1.2.1
%\section{Problem Description.}\label{problemdescription} %% 2.

% Text of your paper here

\section{Introduction}
End-stage renal disease (ESRD), the final stage of chronic kidney disease (CKD), occurs when the kidneys can no longer perform their essential task of removing waste products from the blood. Patients with ESRD require one of two interventions to stay alive: dialysis or kidney transplantation. Dialysis refers to the removal of waste and excess water from the body by circulating blood through a filter surrounded by clean fluid.  While kidney transplantation yields better patient outcomes \citep{Port1993, Schnuelle1998}, the demand for organs far outstrips the available supply, and nearly 100,000 patients await a kidney transplant in the US \citep{UNOS.ORG}.  Furthermore, CKD is highly prevalent among the elderly, and as a result many patients with ESRD are not candidates for a kidney transplant due to age and co-existing medical conditions.   Therefore, dialysis is the only realistic treatment option for the majority of patients with ESRD.  

Hemodialysis (HD) is the most common form of dialysis, accounting for 92\% of the incident dialysis cases in 2011 \citep{atlas}.  HD involves the circulation of blood from a patient through a dialysis machine.  The blood stream is typically accessed in one of two ways: by creation of an \emph{arteriovenous fistula} (AVF) or by insertion of a \emph{central venous catheter} (CVC).  An AVF is created by a surgical procedure in which an artery is connected to a vein in the lower or upper arm.  In contrast, placing a CVC is a minor procedure in which synthetic tubing is inserted directly into a large vein, usually in the neck. The AVF is often considered the gold standard for vascular access \citep{NKF2006} because it is associated with lower infection and mortality rates \citep{DhingraMortality2001, perl_hemodialysis_2011} and higher quality of life \citep{Goro, Wasse,Lopez}. The preference for using AVFs for HD is underscored by the \textit{Fistula First Breakthrough Initiative (FFBI)}, whose mission is ``to improve the survival and quality of life of hemodialysis patients by optimizing vascular access selection - which for most patients will be an AV fistula \ldots'' \citep{FFirst}.  Current guidelines reflect this by suggesting that patients on HD should be referred for an AVF when possible \citep{CSN2006,NKF2006}.

Although the benefits of an AVF over a CVC may seem clear, there are some major differences between them that deserve careful consideration before recommending one access versus the other.  First, a CVC can be used immediately after placement for HD, whereas an AVF requires a lead time of approximately 3 months from the time of surgical creation until it has matured for possible use in HD \citep{rayner, Ethier}.  This is the time it takes for the vein used in the AVF to become thick and large enough to support the insertion of needles necessary for each HD session.  Unfortunately, a majority of created AVFs (around 50\%) do not mature to a point they can be used for HD \citep{Feldman, Lok, Peterson}.  In these cases, patients and their doctors may decide to undergo a subsequent AVF surgery, provided there are still suitable vessels located elsewhere on the arms to allow for this (typically a maximum of three AVF creations may be considered \citep{AVFpaper}). Even if AVF creation is successful, a mature and functional AVF has a limited lifetime, with a 15\% annual failure probability \citep{Roy, Radoui}. Finally, while an AVF has quality of life and morbidity advantages relative to a CVC once it is in use for HD, it still has several disadvantages associated with it prior to that time.  Since the procedure is more invasive than a CVC insertion, it brings about the usual concerns with any surgery (e.g., patient anxiety, infection, post-operative recovery). In some cases, an AVF creation might compromise the blood supply to the hand, which can lead to permanent tissue and neurological damage.  Furthermore, AVFs impose physical limitations (e.g., heavy lifting with the AVF arm is not advised), and some patients find AVFs disfiguring.

The renal  community has recently begun debating the complexities of vascular access choice \citep{Ohare}, raising concerns about whether ``fistula first'' should continue to be the treatment paradigm for all patients. \cite{proAVF2012} and \cite{conAVf2012} discuss opposing views regarding whether or not AVF is the best vascular access for HD patients, and an editorial comments on this debate \citep{editorialAVF12}.  The decision is especially germane for the elderly population; \cite{moist2012optimal} suggests considering factors such as an elderly patient's remaining life expectancy and personal preferences when making a recommendation of vascular access.  This relates to the growing momentum in the medical community to take a personalized and shared approach (between clinicians and patients) to medical decisions, rather than the one-size-fits-all approach of most clinical guidelines \citep{Barry12}. The need for individualized renal care has also been emphasized in  \cite{tamura2011optimizing}.  

The goal of our paper is to bring a data-driven, analytical approach to investigate if or when HD patients should  be referred for AVF surgery. In the spirit of patient-centered care, we focus on the patient's perspective and consider objectives related to patient lifetime and quality-adjusted lifetime.  Our study is also in line with the recent emphasis in the US on the use of comparative effectiveness research (CER) for guiding evidence-based decision making in medicine \citep{Sox09}.  The importance of CER for guiding renal disease treatments in particular is discussed in \cite{Boulware13}.  On a similar note, \cite{moist2012optimal} noted the importance of future quantitative studies evaluating timing and type of vascular access to improve mortality and quality of life in elderly patients.  Our work provides decision makers with both high level insights on AVF vs. CVC decisions as well as quantitative studies to guide decisions specifically for different patient types (including the elderly).   

Our paper is outlined as follows.  We begin with a literature review and outline our contributions in Section \ref{sec:LitRev}.  Section \ref{sec:ModFrame} provides our modeling framework and assumptions.  Section \ref{sec:AnalyticalRes} contains our key analytical results. In Section \ref{sec:num}, we apply our analytical model using data-driven parameters from the literature. Finally, we provide concluding remarks in Section \ref{sec:conc}.


\section{Literature Review} \label{sec:LitRev}
In this section, we review existing literature related to our research in two categories: 1. Operations Research papers on the optimal timing of medical interventions, and 2. clinical papers describing a decision-analytic model of vascular access choice for renal disease patients.

\subsection{Optimal Timing of Medical Interventions}
Decisions regarding the optimal time to apply a medical treatment or screen patients for some disease have received growing attention in the Operation Research/Management Science (OR/MS) community in the past decade. For instance, a number of papers have considered optimal screening policies for detecting breast cancer. \cite{maillart_assessing_2008} developed a partially observed Markov chain model to generate the set of efficient policies, among a selection of easy-to-implement breast cancer screening policies, based on the trade-off between a breast cancer related mortality risk metric and expected mammogram count over a woman's lifetime. \cite{Chhatwal} formulated the problem of breast biopsy timing as a finite-horizon discrete-time Markov decision process (MDP). The state in their MDP model represented the patient's risk score of breast-cancer, determined by a mammography or based on a radiologist's opinion, and actions were limited to either a biopsy or a mammography in the following year. They provided sufficient conditions which ensure the existence of a control-limit type policy that maximizes a patient's quality-adjusted life expectancy. \cite{Turgay} incorporated two methods of detection (self and screen) into a partially observable Markov chain model to form a personalized optimal breast cancer screening policy based on individual risk factors. They proved several structural properties of their proposed model, which helped to solve their problem optimally.

Stochastic optimization models have been used to help with the timing of medical intervention in other contexts as well. For instance, \cite{alagoz_optimal_2004} developed a Markov decision model to investigate the optimal timing of a living-donor liver transplant to maximize the patient's QALE. Patient's health comprised the state of the process, and the actions were either to transplant from a living-donor or wait. They proved sufficient conditions under which the optimal policy is of a control-limit type, i.e., the optimal decision is to transplant when the patient's health is worse than a threshold value and is to wait, otherwise. In an extension to their previous model, \cite{alagoz_choosing_2007} addressed the question of choosing among living-donor and cadaveric livers for an end stage liver disease patient who is offered a liver. \cite{shechter_optimal_2008} addressed the question of when to initiate HIV treatment so as to maximize the expected lifetime or quality-adjusted lifetime of a patient. Results suggested that immediate treatment is optimal, contrary to existing practice at the time of publication. \cite{lee_optimal_2008} used a simulation-based approximate dynamic programming algorithm to derive near optimal strategies for initiation and management of dialysis therapy. They considered the problem of timing of dialysis initiation, whereas we focus on which vascular access to choose for patients already on HD.
 
\subsection{Vascular Access Choice}
The question of vascular access choice and whether or not to refer a patient for AVF arises at various stages of CKD: before a CKD patient begins HD, at the time of HD initiation, and during the course of a patient's dependence on HD. For instance, for an ESRD patient to begin HD with an AVF, an AVF referral should be made well in advance of HD initiation due to the AVF maturation lead-time and surgical failures. Also, a patient without an established or maturing AVF access may need a referral at the time of HD initiation. In the third case, patients on HD who have an AVF fail also face the decision of whether to have another AVF created. 

A number of decision analytic models related to AVF decision making have appeared in the recent clinical literature.  Using a data-driven Monte-Carlo simulation model, \cite{AVFpaper} investigated policies of AVF referral for CKD patients. They formalized two types of policies existing in the literature: 1. threshold strategies in which AVF referral occurs when the estimated glomerular filtration rate of the patient falls below a pre-specified threshold, 2. preparation-window strategies where AVF referral occurs when the anticipated HD initiation date is within the chosen time window. They assessed these policies over a range of values in terms of patient expected lifetime, proportion of AVF incident HD patients, and proportion of unused AVFs. \cite{Hiremath} compared two AVF referral policies for a 70-year-old patient with stage 4 CKD using a Markov model and reported life expectancy and quality adjusted life expectancy as the outcomes. They recommended further research on patient preference and cost implications when making AVF referral recommendations. \cite{Xue} developed a Markov model to study cost-effectiveness of different vascular access alternatives among incident HD patients. They found that the decision of whether to use AVFs or arteriovenous grafts (AVGs), another type of vascular access used in HD, for patients with incident HD depends highly on the AVF maturation failure probability, and they suggested taking this into account for individualized access planning. \cite{Leermakers} also performed a similar cost-effectiveness analysis study and found that AVFs are more cost-effective than AVGs, and motivated initiatives to reduce AVF maturation failure to further improve AVF effectiveness.

\subsection{Contributions}
The purpose of this paper is to address the following questions: 1. Whether new HD patients should be referred for an AVF or not, and 2. Whether existing HD patients should be referred for an AVF if a previous AVF fails. We aim to find individualized optimal policies that maximize a patient's probability of survival and remaining quality adjusted life expectancy, and we consider how referral policies depend on patient age. To our knowledge, this is the first paper to establish analytical results regarding optimal AVF referral policies for HD patients.

Existing recommendations for vascular access choice for HD-dependent patients are scant and in most cases not evidence-based. We construct an analytical, data-driven model that incorporates several key factors when making AVF referral decisions. In particular, patient age, AVF success probabilities, hazard rate functions for patient survival on an AVF vs. CVC, and patient quality of life measures are important drivers of our model-based recommendations.

One of the key model components in determining the optimal policy, the AVF creation disutility, may be difficult to estimate and varies from patient to patient. To circumvent this issue, we introduce a dual view of the optimal policy by using the notion of a critical disutility. We prove that at each decision point, the nephrologist needs to know only if a patient's AVF creation disutility is below or above a critical factor, rather than its exact value, to make the optimal decision. This involves engaging patients in the decision making process, by assessing their individual tolerances for undergoing surgery.

Several unique features of our research contribute to the OR/MS literature on medical decision making. We model a patient's lifetime as a continuous random variable, which facilitates our consideration of a patient's treatment-based non-stationary mortality rate. Also, to our knowledge, our research is the first decision model to consider treatment options which require a stochastic lead-time before they are effective. Whereas the previous models can assume a mammography, transplantation, or HIV treatment can be administered whenever it is desired, an AVF cannot be created instantaneously.  Moreover, there is uncertainty regarding if or when a successful AVF will be attained.  This brings an interesting dynamic to the decision, because the benefit of the AVF may not be as substantial at the time it is ready, and moreover, the patient may die beforehand.

Age-based optimal policies have been considered in some of the existing medical decision problems in the OR/MS literature (e.g., see \cite{Chhatwal, Turgay}). In addition to patient age, we consider duration on HD, as both age and HD duration have been shown to affect patient survival \citep{KurellaSurv, perl_hemodialysis_2011}. For instance, we show that the  decision to perform an AVF surgery for a 75 year old patient who has been dialyzing for five years is different than a 75 year old who has just started HD. 

Finally, unlike the existing literature in which policies are primarily compared based on the expected value of the metric of interest (for instance patient's expected lifetime), we are able to establish that immediate referral for AVF stochastically maximizes (in the first order sense) a patient's lifetime.  This result is interesting and practical because it means that this policy maximizes the probability that a patient survives until a new treatment may become available (e.g., a kidney donation for an HD-dependent patient).

\section{Modeling Framework} \label{sec:ModFrame}
We consider an ESRD patient already on HD with at least one unused AVF opportunity. Note that our model will answer two types of AVF referral questions: 1. should patients who just begin HD be referred for an AVF (assuming no AVF is already in process), and 2. should patients who have an AVF fail during the course of HD be referred for an AVF?  We assume that kidney transplantation is not a realistic option for the patient and thus will depend on HD until death. Also, we assume that the patient has to choose between two vascular access types: CVC, and AVF. We do not consider AVGs here. We discuss the role of AVGs and the possibility of kidney transplantation further in Section \ref{sec:conc}. In Figure \ref{fig:decision}, the decision making framework is illustrated. As the decision flowchart suggests, we have made the following assumption:

\begin{assumption} [Decision points] \label{ass:dec}
A patient can be referred for an AVF at any time, provided there are remaining AVF opportunities and an AVF is not under preparation or being used.
\end{assumption}
Although it might be optimal to refer for a new AVF when the one being used is approaching the end of its lifetime, this is not done in practice, and thus we do not consider it here.

\begin{figure}[htbp]
\centering
\includegraphics[scale=.8]{./files/model_framework.pdf}
\caption{Modeling framework for vascular access dynamics (including decisions and events) for an HD-dependent patient }
\label{fig:decision}
\end{figure}
The dynamics and principles of the model can be summarized as follows. A patient receives HD via an AVF as long as they have an established one. When there is no functional AVF (either when one fails or at the beginning of HD when the patient starts HD without a functional AVF) the patient dialyzes via a CVC as a bridge access. During this time, the policy determines \textit{\textbf{whether}} and \textit{\textbf{when}} to refer a patient for an AVF. If the policy recommends an AVF referral, the patient goes through the AVF creation process and waits until possibly attaining a functional AVF. If all AVF opportunities have been used up, or the policy recommends no further AVF referrals, the patient remains on HD with a CVC until death.

We discuss factors impacting the decision of whether and when to use AVF opportunities in the following sections.
\subsection{Access-Based Patient Survival}
Patient survival on HD depends on the vascular access being used \citep{Astor, KurellaSurv, perl_hemodialysis_2011}. Figure \ref{fig:survival_a}, obtained from \cite{perl_hemodialysis_2011}, shows that patients receiving HD via an AVF experience stochastically better survival than those who receive it via a CVC. Nevertheless, the survival benefit of AVF over CVC, measured by the failure rate difference, diminishes as a patient continues using HD (see Figure \ref{fig:survival_b}, derived from Figure \ref{fig:survival_a}). In addition, a patient's failure rate on either access types increases as the HD duration increases.

\begin{figure}[htbp]
  \centering
  \label{fig:survival}
  \subfloat[Access-based survival probability.]{\label{fig:survival_a}\includegraphics[width=0.5\textwidth]{./files/surv_perl.pdf}}
  \subfloat[Access-based failure rate.]{\label{fig:survival_b}\includegraphics[width=0.5\textwidth]{./files/hrate_perl.pdf}}
\caption{Access-based survival probability and failure rate for a 67 year old HD patient. Figure \ref{fig:survival_a} is obtained from \cite{perl_hemodialysis_2011}, and Figure \ref{fig:survival_b} is derived from Figure \ref{fig:survival_a}.}
\end{figure}
We use these data-driven observations to justify further assumptions below. First, we describe some notation:

\begin{itemize}
\item $t$: time since the patient started HD
\item $\surv{X}{t}$: survival probability function of a random variable $X$ until time $t$ ($\surv{X}{t}=\pr [X > t]$) 
\item $\pdf{X}{t}$: probability density function of a random variable $X$ at time $t$
\item $\hrate{X}{t}$: hazard rate function of a random variable $X$ at time $t$ 
\item $X_t$: residual lifetime of a random variable $X$ at time $t$ (a random variable denoting the remaining lifetime of $X$ from time $t$ onward conditional on survival until time $t$)
\item $\mu(t) \in \{{a, c}\}$: patient's HD access type at time $t$ ($a$ if it is an AVF, and $c$, if it is a CVC).
\item $C$: random variable denoting patient's lifetime when remaining on a CVC from HD initiation time until death.
\item $A$: random variable denoting patient's lifetime when remaining on an AVF from HD initiation time until death.
\item $L$: random variable denoting patient's lifetime
\end{itemize}
Note that the distributions of $C$ and $A$ are dependent on a patient's age at the time HD commences, but we do not denote this dependency for ease of notation.

The following are the definitions for common types of stochastic orders for random variables.
\begin{definition} [Usual stochastic order]
We say $X \le_{st} Y$, if and only if
\begin{align*} %\label{eq:stdef}
\surv{X}{t} \le \surv{Y}{t}: \forall t.
\end{align*}
\end{definition}


\begin{definition} [Hazard rate order]
We say $X \le_{hr} Y$, if and only if
\begin{align*}
\hrate{Y}{t} \le \hrate{X}{t}: \forall t.
\end{align*}
\end{definition}


\begin{definition} [Equality in distribution]
We say random variables $X$ and $Y$ are equal in distribution and denote that by $X \overset{d}{=} Y$ if and only if
\begin{align*}
\surv{X}{t} = \surv{Y}{t}: \forall t.
\end{align*}
\end{definition}


 Our next assumption describes how survival depends on HD duration and vascular access type.
\begin{assumption} [Survival distribution] \label{ass:surv}
 A patient's remaining survival depends on the length of time that the patient has been on HD and the ongoing mode of HD access (an AVF or a CVC), and is independent of the history of HD access type.
 \end{assumption}
 Mathematically, Assumption \ref{ass:surv} indicates that if the patient remains on the same access from $t$ until $t + x$ (for any $x \ge 0$), we have:
 \begin{align}
 &\pr \left ( L_t \ge x \big|  \mu(t')  \text{ for all } t' \le t, \mu(s)={a} \text{ for all } t \le s \le x+t \right) =\surv{A_t}{x}, \label{eq:sAVF}\\
 &\pr  \left (L_t \ge x \big|  \mu(t')  \text{ for all } t' \le t, \mu(s)={c} \text{ for all } t \le s \le x+t \right)=\surv{C_t}{x}. \label{eq:sCVC}
 \end{align}

Equations \ref{eq:sAVF} and \ref{eq:sCVC} indicate that a patient's remaining survival probability is independent of HD access history until time $t$. This Markovian assumption is made for the sake of modeling tractability, and also because there is no data-driven support for an access history dependent consideration of survival.

The following three assumptions formalize the data-driven observations of Figure \ref{fig:survival_b}.
\begin{assumption} [Relative performance]  \label{ass:relative}
The hazard rate of $C$ is higher than or equal to the hazard rate of $A$, at all ages. Mathematically, we have:
\begin{align*} 
\hrate{C}{t} \ge \hrate{A}{t} , \forall t.
\end{align*}
\end{assumption}
Note that Assumption \ref{ass:relative} corresponds to the CVC hazard rate curve lying above the AVF hazard rate curve in Figure \ref{fig:survival_b}, and is equivalent to $ C \le_{hr} A$ by definition. 
Interestingly, by Lemma \ref{lem:hr_eq} of Appendix \ref{sec:supp}, we have that Assumption \ref{ass:relative} is equivalent to the residual lifetime of $C$ is stochastically smaller than the residual lifetime of $A$ at all ages, i.e., we have $C_t \le_{st} A_t , \forall t$ . Throughout this paper, by ``decreasing'' (``increasing'') we mean ``non-increasing'' (``non-decreasing''), unless ``strictly'' is noted.


\begin{assumption} [Diminishing difference] \label{ass:converging}
The difference between hazard rates of $C$ and $A$ decreases in time, i.e., $\hrate{C}{t} - \hrate{A}{t}$ is decreasing in $t$.  
%\begin{align*}
%\hrate{C}{t} - \hrate{A}{t} \downarrow t.
%\end{align*}
\end{assumption}

Note that Assumption 4 corresponds to the diminishing gap between the CVC hazard rate curve and the AVF hazard rate curve of Figure \ref{fig:survival_b}.

Finally, the following assumption states that an HD patient's mortality rate, on either access type, increases with patient age (or rather, we should more precisely say with ``duration on HD'').
\begin{assumption} [Diminishing performance] \label{ass:IFR}
Random variables $A$ and $C$ have the increasing failure rate (IFR) property, i.e., $\hrate{A}{t}$ and $\hrate{C}{t}$ are increasing in $t$.
\end{assumption}
Assumption 5 is demonstrated by the fact that both curves in Figure \ref{fig:survival_b} are increasing.  

\subsection{AVF Creation Process}
After a patient and her clinician decide to use an AVF for HD, she visits a vascular surgeon for AVF placement. After the surgery is performed, the AVF maturation, a process by which a fistula becomes suitable to use for HD, begins (e.g., develops adequate flow, wall thickness, and diameter). It takes approximately 3 months of AVF maturation to learn whether the AVF is usable or not for HD.  However, a major issue for AVF placement is that around 50\% of AVFs fail to mature \citep{Dember2, Dember,Hakim, Xue}. Furthermore, even if an AVF creation is successful, it has an annual failure probability of 15\% \citep{Roy, Radoui}.  These factors are critical to the decision of whether or not a patient should be referred for an AVF.

We use the following notation for random variables describing the AVF creation process:
\begin{itemize}
\item $M_{i}(t_{\text{AVF}_i})$: random variable denoting the maturation time of the $i^\text{th}$ AVF which was created at time $t_{\text{AVF}_i}$.
\item $K_{i}(t_{\text{AVF}_i})$: random variable denoting the total lifetime of the $i^\text{th}$ AVF  which was created at time $t_{\text{AVF}_i}$ (if AVF creation is unsuccessful, then $K_{i}(t_{\text{AVF}_i})=0$).
\end{itemize}

We make the following assumption about the AVF creation process.
\begin{assumption} [AVF maturation and lifetime] \label{ass:AVFs}
All respective random variables describing the AVF creation process, i.e., $M_{i}(t_{\text{AVF}_i}),K_{i}(t_{\text{AVF}_i})$ are identically distributed and stationary, i.e., $\forall i,t_{\text{AVF}_i}:  M_{i}(t_{\text{AVF}_i}) \overset{d}{=} M \text{ and }  K_{i}(t_{\text{AVF}_i})\overset{d}{=}  K$ for some random variables $M$ and $K$. Furthermore, they are all independent of each other and of the patient's survival process.
\end{assumption}
The stationarity assumption is justified by the relatively short life expectancy of HD patients (on average 6.2 years \citep{atlas}). We discuss the independence assumption further in Section \ref{sec:conc}.

Henceforth, we denote $M_{i}(t_{\text{AVF}_i}),K_{i}(t_{\text{AVF}_i})$ by $M_i$ and $K_i$, respectively. The time at which the $i^\text{th}$ AVF was created, $t_{\text{AVF}_i}$, will be clear from the context. 
\subsection{Objective Functions}
\subsubsection{Total Lifetime}
A natural metric for comparing policies is the total lifetime of a patient. Thus, we consider maximizing a patient's total lifetime as one of the objective functions. 

\subsubsection{Quality Adjusted Life Expectancy (QALE)} 
Using AVF for HD not only brings better survival, but also has a slightly higher quality of life for the patient, in comparison with HD using a CVC \citep{Goro, Lopez}. Nevertheless, the process of AVF creation has some disutility associated with it, which can be attributed to the surgery and post-surgery inconveniences, complications or costs. We define a patient's quality adjusted life expectancy as the quality adjusted lifetime on each vascular access minus the AVF surgery disutility for each AVF surgery performed (whether successful or not).

The following  are used in defining the patient's QALE:
\begin{itemize}
\item $q_a$, $q_c$: utility of being an HD patient who receives HD via an AVF or a CVC, respectively.
\item $d$: AVF creation disutility
\item $L_A^{\pi}$, $L_C^{\pi}$: random variables denoting patient's aggregate lifetime on AVF and CVC, respectively, from HD initiation until death, under an arbitrary AVF referral policy $\pi$
\item $N^{\pi}$: random variable denoting number of AVF surgeries performed, under an AVF referral policy $\pi$
\item $Q^{\pi}$: random variable denoting the patient's total QALE, under an AVF referral policy $\pi$
\end{itemize}

Using this notation, we have the following equation for a patient's total QALE under an arbitrary AVF referral policy $\pi$:
\begin{align*}
Q^{\pi}=q_a L_A^{\pi}+q_c L_C^{\pi}-dN^{\pi}
\end{align*}
Based on the estimates in the literature \citep{Goro,Lopez,Wasse}, we make the following assumption about the access-based quality of life coefficients.
\begin{assumption}[Relative quality of life]\label{ass:qol} 
Patients experience a better quality of life dialyzing via an AVF than via a CVC, i.e., we assume $q_a \ge q_c$.
\end{assumption}
\subsection{Dynamic Programming Formulation}
To explain the dynamics of the model to optimize a patient's QALE and prove the results, we formalize the decision making process with a dynamic programming model. The model components are as follows:

\begin{itemize}
\item \textbf{States}: The set $\big \{(t,n), \forall t,n\ge 0 \big\} \cup \big\{\Delta\big\}$ defines the state space. The set of vectors $(t,n)$ consisting of $t$, the time, and $n$ the number of AVF chances left, corresponds to a living state, and the absorbing state $\Delta$ corresponds to the death state. Based on Assumption \ref{ass:dec}, we only need to consider transitions between ``decision states'' (i.e., the subset of $\{(t,n)|n>=1\}$ for which the patient does not have a functional or maturing AVF but has AVF opportunities remaining), the first transition to state $(t,0)$, and the transition to state $\Delta$. 

 we only consider patient's states at which she doesn't have a functional or maturing AVF.

\item \textbf{Actions}: At each state $(t,n \ge 1) $, one of two actions can be taken: either to refer the patient for an AVF at time $t+y$, denoted by $@_y$, or to never refer the patient for an AVF. Note that the never refer action is the case of referral at $y=\infty$. Nevertheless, we keep it in the action space for clarity. When $n=0$, the only option is to remain on CVC for the remainder of the patient's lifetime.  

\item \textbf{Transition probabilities}: From decision state $(t,n|n>=1)$, the patient transitions to state $\Delta$ or to $(t',n-1)$ for some $t' \ge t$. Let $L(t,n,y)$ be the random variable denoting the patient's residual lifetime at time $t$ when  referred at $t+y$ for the current AVF chance and then the optimal policy for the subsequent decisions. The patient transitions to the state $(t'(y)=t+y+M_n+K_n,n-1)$ if $L(t,n,y) \ge y+M_n+K_n$, and to the state $\Delta$, otherwise. Note that the residual lifetime, $L(t,n,y)$, is dependent on $M_n,K_n$ as well.  From state $(t,n|n=0)$, the patient transitions to state $\Delta$.

\begin{figure}[ht]
\centering
\includegraphics[scale=0.6]{./files/dp.pdf}
\caption{ \boldmath Dynamic programming diagram. The diagram shows the dynamics of decisions made about possible AVF surgeries for a patient at time $t$. It shows a scenario in which for the current AVF chance a surgery is planned at time $t+y$, and depending on patient's survival (determined by the random variable $L(t,n,y)$), a future decision for the subsequent AVF chance is due at time $t'$, which itself is determined by random variables $M$ and $K$. }
\label{fig:dp}
\end{figure}

\item \textbf{Immediate reward}: The immediate reward consists of a patient's QALE from time $t$ until the next living state or death. Define the immediate reward as follows (see Figure \ref{fig:dp}):
\begin{align*}
r \big((t,n),y|M=m,K=k,L(t,n,y)=l\big)=\begin{cases}
q_cl& l \le y\\
-d+q_c l& y \le l \le y+m\\
-d+q_c(y+m)+q_a\big(l-(y+m)\big) & y+m \le l \le y+m+k\\
-d+q_c(y+m)+q_ak & y+m+k \le l
\end{cases}
\end{align*}
Depending on the realization of $L(t,n,y)$ (defined above), the patient receives a different immediate reward. In the first case, she dies before the AVF referral (i.e., $l \le y$) and receives $q_cl$ because she is on a CVC during her residual lifetime (from $t$ until $t+l$). In all other cases ($l > y$), she survives until the AVF referral and thus the AVF creation disutility is subtracted. In the second case, the patient dies before the new AVF matures (i.e., $l \le y+m$) and therefore receives $-d+q_cl$ because she is entirely on a CVC from $t$ until $t+l$. In the third case, she survives until the AVF maturation is over but dies before the AVF expires  (i.e., $ y+m \le l \le y+m+k$) and thus receives $-d+q_c(y+m)+q_a\big(l-(y+m)\big)$ because she is on a CVC for $y+m$ and on an AVF for $l-(y+m)$. In the first three cases, the patient transitions to the death state $\Delta$.  In the fourth case, in which  the patient survives until the AVF expires (i.e., $y+m+k \le l$), she transitions to a living state $(t'(y),n-1)$, at which a referral decision is to be made again. In this case, the patient receives $-d+q_c(y+m)+q_ak$ because she is on a CVC for $y+m$ and on an AVF for $k$. If $n=0$, the patient has no AVF chances left and has to remain on a CVC until she dies and thus receives an immediate reward of $q_C l$ (this belongs to the first case since we can equivalently assume $y=\infty$).

Taking expected values over all sample paths, we can define the expected immediate reward of action $@_y$ as: $$R\big((t,n),@_y\big):=\Ex_{M,K,L(t,n,y)} \big[r \big((t,n),y\big|M,K,L(t,n,y)\big)\big].$$
\item \textbf{Policy}: A policy in this problem is a function which maps each state ($t,n$) to an optimal action, being an AVF referral at $t+y$ for some $y$ or to never refer. Note that because of the Markovian properties of the model, we only need to focus on deterministic Markov policies \citep{Puterman}.
\item \textbf{Optimality equation}:\\ We use the following notations in defining the optimality conditions:
\begin{itemize}
\item $v^\pi(t,n)$: the value function (the remaining QALE of a patient) at state $(t,n)$ under an arbitrary policy $\pi$
\item $v(t,n,y)$: the value function of the policy consisting of referral at $t+y$ for the current AVF chance and then the optimal policy for the subsequent decisions.
\item $v(t,n)$: the optimal value function at state $(t,n)$.
\end{itemize}
 The Bellman optimality equation is as follows:
\begin{numcases}{v(t,n)=} \label{eq:suprem}
\max \{ \sup_y v(t,n,y), q_c\Ex C_t \} & $n\ge 1$,  \\ \label{eq:whenN=0}
q_c\Ex C_t & $n=0$. 
\end{numcases}
in which 
\begin{align} \label{eq:vtny}
 v(t,n,y)= R\big((t,n),@_y\big)+\Ex [v(s')],
\end{align}
\end{itemize}
where $s'$ is the patient's future state. We can explain the optimality equation as follows. At each living state in which $n\ge1$, we may decide to refer for AVF at $t+y$ for some $y\ge0$ or never refer for AVF. 
When we decide  not to refer for AVF, which is the only permissible action when $n=0$, the patient remains on a CVC until she dies, and since her residual lifetime under this policy is $C_t$, she receives $q_c\Ex [C_t]$ (Equation \ref{eq:whenN=0}).
If we decide to take the action $@_y$ for the current AVF chance and then the optimal policy for the subsequent decisions, then the value of such policy, denoted by $v(t,n,y)$, consists of the immediate reward $R\big((t,n),@_y\big)$, and the expected future reward $\Ex [v(s')]$ (See Equation \ref{eq:vtny}). Note that the distribution of the future state depends on $t$ and $y$ (see the discussion for immediate reward). Note that we have $v(\Delta)=0$, as the patient receives no more reward in the death state.



\section{Analytical Results} \label{sec:AnalyticalRes}
In this section, we present analytical results. In Sections \ref{sec:optTL} and \ref{sec:QALEGP}, we prove structural properties of the optimal policies for the total lifetime and QALE objectives, respectively. In Section \ref{sec:critdis}, we introduce an alternative view of the optimal policy for the QALE metric. All of the proofs for the analytical results are given in Appendix \ref{sec:AppendixAnal}. Appendix \ref{sec:supp} provides some general results that are used in Appendix \ref{sec:AppendixAnal}.
 
\subsection{Optimal Policy: Total Lifetime} \label{sec:optTL}
Since the survival benefit of an AVF over a CVC decreases as the patient ages, one may think an HD patient should be referred for AVF as soon as an opportunity becomes available, rather than keeping the opportunity for later years.  We prove this below in a stochastic ordering sense: an identical patient referred for AVF earlier than another patient lives stochastically longer than that patient. Of course, this also means that the first patient has a longer expected lifetime.

\begin{theorem} \label{thm:total}
Under Assumptions \ref{ass:dec}-\ref{ass:converging} and \ref{ass:AVFs}, delaying AVF referral stochastically decreases a patient's lifetime.
\end{theorem}
Note that this theorem implies that the optimal policy to maximize the probability of survival until any time $t \ge 0$ (and as a result to maximize expected lifetime) is to refer a patient on CVC for an AVF as soon as possible, provided AVF opportunities remain and no AVF is already in preparation. Also, note that the stochastic ordering result means that the immediate referral policy maximizes the chance a patient may survive until a kidney transplant, the gold standard of care for ESRD patients, becomes available.

\subsection{Optimal Policy: QALE} \label{sec:QALEGP}
Since the survival benefit of AVF decreases as the patient ages, one would want to avoid the AVF creation disutility if it cannot be compensated by the better survival and quality of life associated with HD on an AVF. We will show that the optimal referral policy to maximize a patient's QALE is of a threshold form: if the patient's HD duration (time since the patient initiated HD) at the time of decision is less than a critical value, i.e., if $t < \tau^*$, then it is optimal to refer the patient for AVF at the time of decision; otherwise, the optimal policy is to use a CVC for the rest of the patient's life.  We will also prove that the critical HD duration is independent of the number of AVF chances remaining.

Let $\pi(\tau)$ denote a threshold policy which refers for AVF immediately if $t < \tau$ and recommends a CVC, otherwise. The following theorem proves the optimality of threshold policies.
\begin{theorem} [Optimality of Threshold Policies] \label{thm:QALE}
Under Assumptions \ref{ass:dec}-\ref{ass:qol}, there is a threshold $\tau^*$ such that the policy $\pi(\tau^*)$ maximizes the QALE of the patient. 
\end{theorem}
Note that $\tau^*$ is independent of $n$, the number of remaining AVF chances. For the simplicity of notation, we do not show the dependency of the threshold on the age at HD initiation, AVF creation disutility, and other model parameters. 

\noindent In the next proposition, we prove that the optimal threshold can be found using a binary search.
\begin{proposition}[Binary Search]\label{prop:binsearch}
An optimal threshold policy can be found using a binary search for $\tau^*$ over $[0,t_{\max}]$, where $t_{\max}$ is a reasonable upper bound for $\tau^*$.
\end{proposition}
We can set $t_{\max}$ equal to the time at which the patient reaches the age of 100 years because patients on HD are never referred for an AVF after that age. In the next proposition, we prove that the critical HD duration is decreasing in the AVF creation disutility.
\begin{proposition}\label{prop:dec_d}
The critical HD duration is decreasing in the AVF creation disutility, $d$.
\end{proposition}

\subsection{Critical Disutility}\label{sec:critdis}
In Section \ref{sec:QALEGP}, we proved that there exists a threshold policy that maximizes a patient's QALE. The result of Theorem \ref{thm:QALE} assumes one already has an estimate of the patient's disutility for an AVF creation. However, this may difficult to estimate precisely in practice.  To circumvent this practical challenge, we introduce a a dual view of the HD duration threshold policy. We show that at any time, the decision of whether to do an AVF surgery or not is determined by comparing the patient's AVF creation disutility with a critical value. Thus, in order to make a decision, we only need to know whether the AVF creation disutility is above the critical value or not, rather than require a precise estimate of the AVF disutility itself.
\begin{theorem} [Critical Disutility] \label{thm:cdis}
For any HD duration $t$, there exists a non-negative critical AVF creation disutility, denoted by $d^{\text{cr}}(t)$, such that the optimal decision at time $t$ is to do an AVF surgery immediately if the patient's AVF creation disutility is less than the critical disutility (i.e., if $d < d^{\text{cr}}(t)$), and is to use CVC for the rest of patient's life, otherwise.
\end{theorem}

Note that if $d=0$, i.e., when there is no disutility for AVF creation, then at any time, the optimal policy to maximize the QALE of a patient is to refer patients for AVF as soon as an opportunity becomes available. Also, letting $q_a=q_c=1$ and $d=0$, we have that QALE equals the life expectancy of the patient. Thus by Theorem \ref{thm:cdis}, the optimal policy to maximize the expected lifetime of the patient is to refer immediately at all HD durations. This coincides with the result of Theorem \ref{thm:total}. Nevertheless, Theorem \ref{thm:total} gives a stronger result for the (unadjusted) lifetime metric, based on stochastic ordering, compared to the QALE comparison of Theorems \ref{thm:QALE}.

Based on the following corollary, we can use the critical disutility function to find the critical HD duration for patients with different values of  AVF creation disutility $d>0$.
\begin{corollary} [Relationship between Critical Disutility and Critical Duration] \label{cor:dcrmap}
Let $\tau^*(d)$ be an optimal HD duration threshold for a patient with AVF creation disutility $d$. Then, if  Assumptions \ref{ass:IFR} and \ref{ass:qol} hold strictly, we have that $d^{cr}(t)$ is strictly decreasing and thus invertible. If in addition we assume that $\lim\limits_{t \to \infty} \hrate{A}{t} =\infty$, then we have
\begin{align*} %\label{eq:equiv}
\tau^*(d)=\begin{cases}
d^{-1}_{cr}(d) &:  0 < d < d^{cr}(0)\\
0&:  d \ge d^{cr}(0)
\end{cases}
\end{align*}
where $d^{-1}_{cr}(d)$ represent the inverse function of $d^{cr}(t)$.
\end{corollary}

Note that Theorem \ref{thm:cdis} provides an alternative way of comparing the optimal policy for individual patients as follows: if the critical disutility for one patient is always smaller than another, then the first patient has a smaller HD duration threshold, given that both patients have the same AVF creation disutility. The following result provides further comparative statics to explore the effect of several model parameters on the critical disutility.
\begin{theorem}\label{thm:compdcrt}
Consider two patients types indexed by 1 and 2 whose characteristics satisfy all of the following properties:
\begin{enumerate}
\item $M^{(2)} \le_{st} M^{(1)}$
\item $K^{(1)} \le_{st} K^{(2)}$
\item $q_c^{(1)} \le q_c^{(2)}$
\item  $q_a^{(1)} - q_c^{(1)} \le q_a^{(2) }- q_c^{(2)}$
\item $C^{(1)} \le_{hr} C^{(2)}$
\item $[\hrate{C^{(1)}}{t}- \hrate{A^{(1)}}{t}] \le [\hrate{C^{(2)}}{t} -\hrate{A^{(2)}}{t}]: \forall t$
\end{enumerate}
where $(i)$ denotes the patient's index. Then, $d^{\text{cr}}_{(1)}(t) \le d^{\text{cr}}_{(2)}(t), \forall t$.
\end{theorem}


In words, Theorem \ref{thm:compdcrt} indicates the following. In terms of the AVF creation process, the critical disutility is higher when the AVF maturation time is (stochastically) shorter, and/or if the AVF lifetime is (stochastically) longer (conditions 1 and 2 respectively). With respect to the QALE parameters, the critical disutility is higher when an HD-dependent patient has a better quality of life (whether via a CVC or an AVF), and/or when the difference of quality of life between an AVF and a CVC is higher (conditions 3 and 4 respectively). A similar argument applies to the patient's survival (conditions 5 and 6), i.e., when a patient has a better HD-based survival and/or a better AVF-based HD survival (compared with a CVC-based survival), then the critical disutility is higher.

Note that the result of Theorem \ref{thm:compdcrt} holds when all of the properties are satisfied at equality except for one, which is a strict inequality. For instance, consider a case in which two patients are identical expect for their AVF maturation, with $M^{(2)} <_{st} M^{(1)}$. Then, based on this theorem we have $d^{\text{cr}}_{(1)}(t) \le d^{\text{cr}}_{(2)}(t), \forall t$.

In the following theorem, we show that the critical disutility is proportional to the AVF creation success probability.
\begin{theorem}\label{thm:pavf} The critical disutility is proportional to the AVF creation success probability.
\end{theorem}
Based on this theorem, the critical disutility can be easily adjusted for different values of a patient's AVF surgery success probability.

\section{Numerical Results}\label{sec:num}
To demonstrate the results of Theorems \ref{thm:QALE} and \ref{thm:cdis}, we performed a numerical study. The baseline values for different parameters of the model and sources used are given in Table \ref{tab:params}. 
\input{baseline.tex}

For patients' HD survival, we used  \cite{perl_hemodialysis_2011}, which provides only the first five years of survival outcomes for a cohort of 67 year old patients. To obtain complete survival curves, we extrapolate the hazard rate functions so that Assumptions \ref{ass:relative}-\ref{ass:IFR} are satisfied. Specifically, we assume that the AVF and CVC hazard rates increase linearly after the last observed hazard rate with slopes $\alpha_A$ and $\alpha_C$, respectively. We need to assume $\alpha_A \ge \alpha_C \ge 0$, so that Assumptions \ref{ass:converging} and \ref{ass:IFR} are satisfied. To have Assumption \ref{ass:relative} met, we modify the hazard rates for CVC such that after the point the hazard rate curves meet (if they ever meet, which is always the case when $\alpha_A > \alpha_C$), we have that $\hrate{C}{t}=\hrate{A}{t}$, with the slope of the line equal to $\alpha_A$ (see Figure 4 for an illustration). We calculated the average rate of increase for the AVF and CVC hazard rate functions (that is the slope connecting first and last observed hazard rates). Denoting these slopes with $\mathbf{\bar{r}}_A$ and $\mathbf{\bar{r}}_C$ respectively, we assumed $\alpha_A=\mathbf{\bar{r}}_A$ and $\alpha_C=\mathbf{\bar{r}}_C$ (below, we perform one-way sensitivity analyses by considering scenarios in which  $\alpha_A=(1\pm 25\%)\mathbf{\bar{r}}_A$ and  $\alpha_C=(1\pm 25\%)\mathbf{\bar{r}}_C$).
% something wrong with Figure 4 labels. I had to manually write 4.
\begin{figure}[h!]\label{fig:basecase}
  \centering \includegraphics[width=0.5\textwidth]{./files/basecase.pdf}
\caption{Base case hazard rate functions for a 67 year old patient's lifetime on-HD.}
\end{figure}
Based on the hazard rate functions, a 67 year old patient's entire survival curve was calculated. We used the result of Theorem \ref{thm:cdis} to calculate the critical disutility as a function of HD-duration using Monte-Carlo simulation (see the proof of Theorem \ref{thm:cdis} in Appendix \ref{sec:AppendixAnal}). Figure \ref{fig:cdis67} shows the critical disutility under the baseline assumption for survival extrapolation. For example, a 67 year old patient who has been on HD for 2 (3) years should be referred for AVF provided her AVF disutility is less than 85 (65) QALE days. Recall that the motivation for the critical disutility approach was for cases in which it might be difficult to estimate precisely a patient's disutility for the AVF surgery.  However, based on Corollary \ref{cor:dcrmap}, Figure \ref{fig:cdis67} can also be inverted to answer questions regarding a patient for whom a precise estimate of the AVF disutility is obtained.  For example, the figure also indicates that if a 67 year old patient has a disutility of 85 (65) QALE days, then she should be referred for AVF as long has she has been on HD less than 2 (3) years. 
\begin{figure}[h!]
  \centering
  \subfloat[Critical disutility and duration for a 67 year old patient.  ]{\label{fig:cdis67}\includegraphics[width=0.5\textwidth]{./files/cdis67.pdf}}
  \subfloat[Critical disutility for 67 and 82 year old patients.]{\label{fig:cdis82}\includegraphics[width=0.5\textwidth]{./files/cdis82.pdf}}
\caption{Critical disutility and HD duration for 67 and 82 year old patients. In Figure \ref{fig:cdis67}, the critical HD duration for 67 year old patients with AVF creation disutility of 65 and 85 QALE days is illustrated. It also shows the critical disutility for a 67 year old who just begins HD is 130 QALE days.}
\end{figure}

Studies show that patients' HD mortality rates on either vascular access types are higher when they start HD at older ages (see \cite{CARVALHO, Astor,  KurellaSurv}). Also, the difference in failure rate between HD on an AVF and a CVC decreases with age of HD onset (see \cite{Astor}). Older patients also have a higher chance for AVF failure (\cite{Peterson, Levin}). Therefore, based on Theorem \ref{thm:compdcrt}, we can deduce that a patient starting HD at higher ages has a smaller critical disutility.

To visualize the impact of age at HD initiation on the critical disutility, we have plotted the critical disutility curves for patients who start HD at ages of 67 and 82 years in Figure \ref{fig:cdis82}. As the plot shows, the critical disutility of the older patient is always smaller. For instance for the time of HD initiation, a 67 year old patient will be referred for AVF as long as her AVF creation disutility is below 130 QALE days, while an 82 year old patient will be referred only when her AVF creation disutility is below 70 QALE days (Figure \ref{fig:cdis82}).

In Figure \ref{fig:QALEcomp}, we plot the \% QALE increase from the non-optimal policy to the optimal policy as a function of the AVF creation disutility for an 82 year old patient with one AVF chance, i.e., for $n=1$. We have compared the two policies of ``no AVF referral'' and ``referral at HD initiation'', as they represent two opposing opinions in the literature \citep{proAVF2012, conAVf2012}, and therefore the figure indicates what can be gained if a decision maker adheres to a suboptimal policy one side of the threshold or the other. For $d < d^{cr}(0)$, the optimal policy is to refer the patient for AVF at the time of HD initiation, whereas for $d \ge d^{cr}(0)$, the optimal policy is to remain on a CVC. Let $\% \Delta(d)$ denote the \% QALE increase. We can find $\% \Delta(d)$ as follows. If we refer the patient for AVF at HD initiation, she gains $v(0,1,0)$, whereas the no-referral policy yields the patient $q_c\Ex C$. For $d <d^{cr}(0)$, using the definition of $d^{cr}(t)$, we have:
\begin{align*}
\% \Delta(d)=\frac{v(0,1,0)-q_c\Ex C }{q_c\Ex C }=\frac{d^{cr}(0)-d}{q_c\Ex C}.
\end{align*}

For $d \ge d^{cr}(0)$, we have:
\begin{align*}
\% \Delta(d)=\frac{q_c\Ex C-v(0,1,0)}{v(0,1,0)}=\frac{d-d^{cr}(0)}{q_c\Ex C+d^{cr}(0)-d}  .
\end{align*}

\begin{figure}[h!]
\centering
\includegraphics[scale=.70]{./files/QALECompare.pdf}
\caption{ \boldmath \% Remaining QALE increase from the non-optimal policy to the optimal policy as a function of the AVF creation disutility for an 82 year old patient with $n=1$. We have compared the two policies of ``no AVF referral'' and ``referral at HD initiation'', with the former being optimal for $d \ge d^{cr}(0)$ and the latter for $d \le d^{cr}(0)$.}
\label{fig:QALEcomp}
\end{figure}

\subsection{Sensitivity Analysis}
We also performed a sensitivity analysis to see how robust the results are to the changes in the input parameters. The parameters and values tested for one-way and two-way sensitivity analyses and the corresponding critical disutilities at the time of HD initiation are given in Table \ref{tab:SA}. For instance, the critical disutilities for patients with 60\% and 20\% chances of success in having a matured AVF are 223 and 76 QALE days, respectively. Since the first patient has a higher chance of surgery success, she benefits from the surgery more than the other patient, and as a result, she should be referred for AVF at the time of HD initiation as long as her surgery disutility is less than 223 QALE days, while the other patient benefits from AVF surgery only when the surgery disutility is less than 76 QALE days. As the results in Table \ref{tab:SA} suggest, the critical disutility is most sensitive to the AVF surgery success probability. Based on Theorem \ref{thm:pavf}, the critical disutility is proportional to this parameter, and therefore, it can be easily adjusted by a nephrologist based on her perception of a patient's AVF surgery success probability or existing statistics in the local practice.
% ---------------------
\begin{table}[h]
\small
  \centering
  \caption{Sensitivity analysis for the critical disutility (QALE days) of a 67 year old HD incident patient computed using Monte-Carlo simulation. The default values for each parameter are given in Table \ref{tab:params}.}
      \label{tab:SA}%
    \begin{tabular}{p{4.5cm}  l  c c c c c}
    \toprule

    Parameter & Value  & Critical disutility\\
	 	 \toprule \toprule 
	    N/A   & Default & 151  \\ \hline

    \multirow{2}[4]{*}{\parbox{4cm}{AVF Surgery Success Probability}} & 0.2   & 76   \\
          & 0.6   & 223    \\
   	\midrule    
    \multirow{2}[4]{*}{\parbox{4cm}{Functional AVF Annual Failure Rate}} & 0.1   & 172   \\
          & 0.2   & 134  \\
   	\midrule    
    \multirow{3}[4]{*}{Maturation Time (months)} & Uniform [3,5] & 150\\
          & Uniform [4,6] & 149    \\
          & Uniform [1,6] & 150   \\ %\hline
   	\midrule              
    \multirow{3}[4]{*}{QALE Coeff [CVC, AVF]} & [0.73,0.81] & 164   \\
          & [0.75,0.81] & 158   \\
          & [0.81,0.81] & 139 \\
        
    \midrule              
    \multirow{4}[4]{*}{\parbox{4cm}{Patient's survival projection parameters [$\alpha_A,\alpha_C$]}} & [$\mathbf{\bar{r}}_A$,$1.25*\mathbf{\bar{r}}_C$] & 151   \\
              & [$\mathbf{\bar{r}}_A$,$0.75*\mathbf{\bar{r}}_C$] & 150   \\
              & [$1.25*\mathbf{\bar{r}}_A$,$\mathbf{\bar{r}}_C$] & 147 \\   
               & [$0.75*\mathbf{\bar{r}}_A$,$\mathbf{\bar{r}}_C$] & 155 \\    
    \bottomrule

    \end{tabular}%
\end{table}
% ---------------------


\section{Conclusion}\label{sec:conc}
In this work, we considered the problem of vascular access choice between a CVC and an AVF for HD patients, with a goal of maximizing a patient's total lifetime and QALE. We analytically proved that delaying AVF referral stochastically decreases a patient's lifetime. As a result, the policy of ``use the next AVF (opportunity) as soon as a patient starts HD or when the one being used fails'' maximizes a patient's survival probability. We also proved that the optimal policy to maximize a patient's QALE is of a threshold type: there is an HD duration threshold before which immediate referral is the optimal choice, while after that time, CVC is the optimal vascular access choice for the remainder of the patient's lifetime.

The AVF creation disutility plays an essential role in determining the critical HD duration of the QALE optimal policy. Since patients may feel differently about the disutility of AVF surgery, and also because it is not an easy parameter to elicit from a patient, our model provides an alternative way to make the optimal referral decision. We showed that the decision of whether to perform an AVF surgery or not can be determined solely by comparing the patient's AVF creation disutility with a boundary value reflecting the prospective additional quality lifetime for the patient, which we refer to as the critical disutility. Thus, a nephrologist can inform the patient of the benefits and inconveniences of undergoing the AVF surgery, and then, they can collectively decide whether to do the surgery or not. Even if a rough estimate of the patient's disutility for AVF surgery indicates that it is clearly below or above the critical disutility, then it will be clear that the patient should or should not, respectively, be referred for an AVF. This also facilitates getting patients involved in the decision making process, one of the key recommendations of the Institute of Medicine's report on patient-centered care, which has been emphasized in the medical community in the past decade \citep{IM}.

Referral policies for the elderly is a particularly important discussion in the nephrology community \citep{KurellaFunc, Vac, tamura2011optimizing}. Limited remaining life expectancy suggests extra careful consideration as to whether such patients should be referred. Although higher mortality rate for the elderly impacts the vascular access decision \citep{tamura2011optimizing}, our results suggest that other key parameters such as lower AVF maturation rates would also imply that many elderly patients may be better off remaining on a  CVC (because their critical disutility will be lower, as shown in Figure \ref{fig:cdis82}). 

A few assumptions pose limitation to our study. We chose not to include AVGs as an alternative vascular access, since the prevalence of AVGs in HD patients has dramatically decreased in the past decade (it has dropped from 65\% to 20\% in the US from 1995 to 2010 \citep{Vassalotti}). Also, AVGs are considered by US nephrologists as an alternative to an AVF primarily for those who have a low chance of developing a mature AVF; i.e., for the elderly or those with a history of failed AVFs \citep{Xi}. We also did not consider alternative renal replacement therapies such as peritoneal dialysis or kidney transplantation. Nevertheless, we proved that following the optimal policy for the total lifetime objective maximizes the probability that a patient survives until some future time when she may switch to another treatment option, including kidney transplantation. We assumed that lifetimes and maturation times of different AVFs created for a patient are independent of  one another, as there is no evidence in the literature suggesting otherwise. We would need to adapt our model to incorporate this extra modeling complexity if future data suggests a strong correlation among these variables.
\newpage
\begin{APPENDICES}
\section{Supplementary Results} \label{sec:supp}
  \label{sec:AppSupp}
 In this appendix, we provide some general results which are used in the subsequent appendix in proving analytical results of the paper.
 \begin{lemma}\label{lem:hr_eq}
 Each of the following are equivalent to $X \le_{hr} Y$:
 \begin{enumerate}
 \item $X_t \le_{st} Y_t, \forall t$
 \item  $\dfrac{\surv{X}{t}}{\surv{Y}{t}}$ is decreasing in $t$.
 \end{enumerate}
 \end{lemma}
  
 \proof{Proof.}
  For (1) see Equation 1.B.7 in \cite{shaked2007stochastic}. For (2), see Theorem 1.3.3 in \cite{muller}. \Halmos
 \endproof
 
 \begin{lemma}[Closure of stochastic order under mixture]\label{lem:pres}
 Let $X$, $Y$, $Z$ be random variables such that for all values of $z$, we have $[X | Z=z] \le_{st} [Y|Z=z]$. Then, $X \le_{st} Y$.
 \end{lemma}
 
 \proof{Proof.}
 See Theorem 1.2.15 in \cite{muller}. \Halmos
  \endproof
 
 \begin{lemma}\label{lem:ass2_a} Assumption \ref{ass:converging} is equivalent to having that $\dfrac{\surv{C}{t}}{\surv{A}{t}}$ is a log-convex function of $t$.
 \end{lemma}
 
 \proof{Proof.}
 Note that $\diff{\ln \surv{X}{t}}{t}=-\hrate{X}{t}$. Since $\diff{\ln \dfrac{\surv{C}{t}}{\surv{A}{t}}}{t}=\diff{\ln \surv{C}{t}}{t}-\diff{\ln \surv{C}{t}}{t}=\hrate{A}{t}-\hrate{C}{t}$, the result follows from Assumption \ref{ass:converging} and the fact that a differentiable function is convex if and only if its derivative is increasing.\Halmos
 \endproof
 
 \begin{lemma}\label{lem:log-conv}
 Assume that $g$ is a differentiable and log-convex function. Then, $\frac{g(x)}{g(x+a)}$ is decreasing in $x$ for any $a \ge 0$.
 \end{lemma}
 \proof {Proof.}  It suffices to show that $\ln\frac{g(x)}{g(x+a)}=\ln g(x)-\ln g(x+a)$ is decreasing in $x$. Define $G:=\ln g$, a convex function by assumption. Since $\diff{\ln\frac{g(x)}{g(x+a)}}{x}=\diff{G(x)}{x}-\diff{G(x+a)}{x} \le 0$, based on the fact that the derivative of a convex function is increasing, we have that $\ln\frac{g(x)}{g(x+a)}$ is decreasing in $x$. \Halmos
  \endproof
  
 \begin{lemma}\label{lem:IFR}
 The random variable $X$ has the IFR property if and only if $X_t$ is stochastically decreasing in $t$. 
 \end{lemma}
 
 \proof {Proof.} ~\\$\rightarrow$ Choose $t \le t'$ and $s \ge 0$ arbitrarily. We have $\hrate{X_{t'}}{s}=\hrate{X}{t'+s}$ and $\hrate{X_t}{s}=\hrate{X}{t+s}$. Since $X$ is IFR, we have $\forall s, \hrate{X_t}{s} \le \hrate{X_{t'}}{s}$. Thus, $X_{t'} \le_{hr} X_t$ by definition which implies $X_{t'} \le_{st} X_t$ (hazard rate order implies the stochastic order (see Lemma \ref{lem:hr_eq})).\\
 $\leftarrow$ Choose $t \le t'$ arbitrarily. For all $s \ge 0$, we have $X_{t'+s} \le_{st} X_{t+s}$ by assumption. By Lemma \ref{lem:hr_eq}, $X_{t'} \le_{hr} X_t$ which implies $\hrate{X_t}{s} \le \hrate{X_{t'}}{s}$. Since
 $\hrate{X_{t}}{s}=\hrate{X}{t+s}$ and $\hrate{X_{t'}}{s}=\hrate{X}{t'+s}$, we have $\hrate{X}{t+s} \le \hrate{X}{t'+s}$ and thus the result.\Halmos
  \endproof
  
 \begin{lemma}\label{lem:res'} If $\surv{X}{t}$ is differentiable, then the mean residual lifetime of a random variable $X$ is differentiable. Moreover, we have:
 $$\diff{\Ex X_t}{t}=\hrate{X}{t}\Ex X_t-1.$$
 \end{lemma}
 \proof {Proof.}  See \cite{gupta2003representing}. \Halmos
  \endproof
 \begin{lemma} \label{lem:fincstorder} Let $f:\mathbf{R}^n \mapsto \mathbf{R}$ be an increasing function and $X_i$ and $Y_i$ be an independent set of random variables with $X_i \le_{st} Y_i$. Then, $$f(X_1,\ldots, X_n) \le _{st} f(Y_1,\ldots, Y_n).$$  
 \end{lemma}
 \proof {Proof.}  See Theorem 1.A.3 in \cite{shaked2007stochastic}.\Halmos
  \endproof
\newpage
 \section{Proofs for Analytical Results}\label{sec:AppendixAnal}
 In this appendix, we provide proofs of the analytical results of the paper.
 % Results appear according to their corresponding section in the manuscript.
 
\subsection{Proof of Theorem \ref{thm:total}}
We first show a preliminary lemma that facilitates proving results about the residual lifetime and the residual QALE. We prove that at any time, the residual lifetime of random variables $A$ and $C$ also have the properties stated in Assumptions \ref{ass:relative}-\ref{ass:IFR}.
\begin{lemma} \label{lem:assgen}
Assumptions \ref{ass:relative}-\ref{ass:IFR} apply to $A_t$, and $C_t$ as well. In other words, for all $ s,t \ge 0 $, we have  $ C_t \le_{hr} A_t $, $ \hrate{C_t}{s}-\hrate{A_t}{s} $ is decreasing in  $ s $, and $ \hrate{A_t}{s} $,  $ \hrate{C_t}{s}$ are increasing  in $s$.
%\begin{align*}
%\forall s,t \ge 0:\hspace{3 mm} C_t \le_{hr} A_t \normalfont{ \text{, }} \hspace{2 mm} \hrate{C_t}{s}-\hrate{A_t}{s} \text{ is decreasing in } s \normalfont{ \text{,  }} \hspace{2 mm} \hrate{A_t}{s}, \hrate{C_t}{s} \text{ are increasing  in } s.
%\end{align*}
\end{lemma}
\proof {Proof.}
The result directly follows from the fact that $\hrate{X_t}{s}=\hrate{X}{t+s}$ for any random variable $X$, and $t,s\ge 0$. \Halmos
\endproof

Now, suppose one could set the AVF use time (rather than referral time) at $t=u$. We prove that the residual lifetime decreases stochastically in $u$. In Theorem \ref{thm:total}, we will show that this result extends to the case of AVF referral time. Before that, we provide further notation that will be used in what follows.
\begin{itemize}
\item $u$: time to use an AVF
\item $L(t,n)$: patient's residual lifetime at time $t$, given $n$ remaining AVF chances, under the optimal policy (one that maximizes patient's survival function, point-wise; if such policy exists)
\item $L(t,n,u)$: patient's residual lifetime at time $t$ when we use the first AVF chance at $t+u$ and follow the optimal policy for the subsequent $n-1$ AVF chances
\end{itemize}

\begin{proposition}\label{prop:total}
Under Assumptions \ref{ass:dec}-\ref{ass:AVFs}, $L(t,n,u_2) \le_{st} L(t,n,u_1)$, whenever $u_1 \le u_2$.
\end{proposition}

\proof {Proof.} 
Let $L(u):=\big[L(t,n,u) \big |K_n=k]$. By closure of stochastic order under mixture (Lemma \ref{lem:pres}), it suffices to prove that for all $k$, $\surv{L(u)}{a}$ is decreasing in $u$ (for all $a$)  . We prove this by induction on $n$.\\
\noindent $\rightarrow$ Base case: $n=1$: \\
Depending on the values of $u, a, k$ we can calculate $\surv{L(u)}{a}$ as follows (see Figure \ref{fig:fig1}):
\begin{figure}[htbp]
\centering
\includegraphics[scale=0.7]{./files/LT-F1.pdf}
\caption{\boldmath Possible cases for $\surv{L(u)}{a}$}
\label{fig:fig1}
\end{figure}
\begin{itemize}
\item Case 1: $a \le u$: We have $\surv{L(u)}{a} \xlongequal{A(\ref{ass:surv})} \pr [C_t>a]=\surv{C_t}{a}$.
\item Case 2: $[a-k]^+ \le u \le a$: We have
\begin{align*}
\surv{L(u)}{a}&=\pr [C_t>u , A_{t+u}>a-u  ]=\pr [C_t>u]\pr[A_{t+u}>a-u \big |C_t>u]\\
& \xlongequal{A(\ref{ass:surv})} \pr [C_t>a]\pr[A_{t+u}>a-u] \xlongequal{A(\ref{ass:surv})}\pr [C_t>u].\pr[A_t>a|A_t>u]=\surv{C_t}{u}\frac{\surv{A_t}{a}}{\surv{A_t}{u}}
\end{align*}
\item Case 3: $0 \le u \le [a-k]^+$: We have:
\begin{align*}
\surv{L(u)}{a}&=\pr [C_t>u , A_{t+u}>a-u  , C_{t+u+k}>a-(u+k) ]\\
&=\pr [C_t>u].\pr[A_{u+t}>k \big |C_t>u].\pr[C_{u+k}>a-(u+k) \big |A_{t+u}>k , C_t>u]\\
&\xlongequal{A(\ref{ass:surv})} \pr [C_t>u].\pr[A_{t}>k+u \big |A_t>u].\pr[C_{t}>a \big |C_{t}>u+k]\\
& \xlongequal{A(\ref{ass:surv})} \surv{C_t}{u}.\frac{\surv{A_t}{k+u}}{\surv{A_t}{u}}\frac{\surv{C_t}{a}}{\surv{C_t}{u+k}}=
\surv{C_t}{a}.\frac{\surv{C_t}{u}}{\surv{A_t}{u}}/\frac{\surv{C_t}{u+k}}{\surv{A_t}{u+k}}
\end{align*}
\end{itemize}
in which $A(n)$ represents implication of Assumption $n$.
Note that $\surv{L(u)}{a}$ is continuous within each range, and its value on the boundary points coincides. Therefore, it suffices to prove that in each range, $\surv{L(u)}{a}$ is decreasing. In Case 1, the function is constant and thus the result holds trivially. In Case 2, since $C_t \le_{hr} A_t$ according to Lemma \ref{lem:assgen}, the function is decreasing using Lemma \ref{lem:hr_eq}. In Case 3, Lemma \ref{lem:assgen} and Lemma \ref{lem:ass2_a} imply that $\frac{\surv{C_t}{u}}{\surv{A_t}{u}}$ is log-convex in $u$. Using Lemma \ref{lem:log-conv}, we have that $\surv{L(u)}{a}$ is decreasing in $u$.

\noindent $\rightarrow$ Induction step: Assume $L(t,n-1,u_2) \le_{st} L(t,n-1,u_1)$, for all $u_1 \le u_2$. We prove that if $u_1 \le u_2$, then $L(t,n,u_2) \le_{st} L(t,n,u_1)$.

Since stochastic order is a partial order, using the transitivity property we can instead prove that $L(u_2) \le_{st} L'$ and $L' \le_{st} L(u_1)$, in which $L'$ is the lifetime under a hypothetical situation similar to $L(u_1)$ with the difference that the decision to use the subsequent AVF is delayed until $u_2+k$ (see Figure \ref{fig:prop1-induction}).
\begin{figure}[htbp]
\centering
\includegraphics[scale=0.70]{./files/LT-F2.pdf}

\caption{\boldmath Induction step and the hypothetical random variable $L'$}
\label{fig:prop1-induction}
\end{figure}
\begin{itemize}
	\item $L(u_2) \le_{st} L'$: For $x \le u_2+k$, we have that $\surv{L(u_2)}{x}=\surv{L(t,1,u_2)}{x}$ and $\surv{L'}{x}=\surv{L(t,1,u_1)}{x}$. Thus the result follows from induction base. Otherwise, we have 
	\begin{align*}
		\surv{L(u_2)}{x}=\surv{L(u_2)}{u_2+k}.& \surv{L(u_2+k,n-1)}{x-[u_2+k]}, \\
		\surv{L'}{x}=\surv{L'}{u_2+k}.& \surv{L(u_2+k,n-1)}{x-[u_2+k]}.
	\end{align*}

	Based on the previous result, we have $\surv{L(u_2)}{u_2+k} \le \surv{L'}{u_2+k}$, and thus we get the result.
	\item $L' \le_{st} L(t,n,u_1)$. For $x \le u_1+k$, we have that $\surv{L(u_1)}{x}=\surv{L'}{x}=\surv{L(t,1,u_1)}{x}$. For $x  \ge u_1+k$, 
		\begin{align*}
			\surv{L(u_1)}{x}=\surv{L'}{u_1+k}.& \surv{L(u_1+k,n-1,0)}{x-[u_1+k]}, \\
			\surv{L'}{x}=\surv{L'}{u_1+k}.& \surv{L(u_1+k,n-1,u_2-u_1)}{x-[u_1+k]}.
		\end{align*}
		Using the induction hypothesis, we have $L(u_1+k,n-1,u_2-u_1) \le_{st} L(u_1+k,n-1,0)$, and thus we have the desired result. \Halmos
\end{itemize} 
\endproof

We now extend the result of Proposition \ref{prop:total} from AVF use time to AVF referral time.

\begin{repeattheorem}[Theorem \ref{thm:total}.]
Under Assumptions \ref{ass:dec}-\ref{ass:converging} and \ref{ass:AVFs}, delaying AVF referral stochastically decreases a patient's lifetime.
\end{repeattheorem}

\proof {Proof.} In Proposition \ref{prop:total}, we proved that the patient's residual lifetime stochastically decreases in AVF use time. Since later referral means later AVF use time, then the patient's lifetime is stochastically decreasing in the referral time, as well. Mathematically, we can prove it as follows. Let $r$ be the referral time, and $L_r$ be the lifetime of the patient when the patient is referred at $r$ for the current AVF, and optimally (with respect to survival function) for the subsequent chances. Note that for AVF use time we have $u=r+M$. Fix $M=m$, arbitrarily. If $r_1\le r_2$, then $u_1 \le u_2$. As a result of Proposition \ref{prop:total}, 
$$\forall w,m:[L_{r_2}\big| M=m] \le_{st} [L_{r_1}\big|M=m].$$ 
Now, by closure of stochastic order under mixture (Lemma \ref{lem:pres}), this implies $L_{r_2} \le_{st} L_{r_1}$. \Halmos
\endproof

\subsection{Proof of Theorem \ref{thm:QALE}} 
We prove the optimality of threshold policies (Theorem \ref{thm:QALE}) in two steps. In the first step, we prove the existence of an optimal HD-duration threshold policy for the case $n=1$ in Proposition \ref{prop:qalen=1}. In the second step, we prove that the same threshold policy is optimal for the case $n>1$, as well. We use Lemmas \ref{lem:delta}-\ref{lem:v1} to prove Proposition \ref{prop:qalen=1}.

Recall that $v(t,n,y)$ is the value function at the state $(t,n)$ when we follow the policy consisting of an AVF referral at $t+y$ for the current AVF chance and then the optimal policy for the subsequent decisions. Let $\pi_0$ denote the policy of using CVC for the rest of the patient's life (hereafter referred to as the ``no-referral'' policy). Under this policy, the patient remains on a CVC until she dies, and since her residual lifetime under this policy is $C_t$, her QALE is $q_c\Ex [C_t]$, i.e., we have $v^{\pi_0}(t,n)=q_c\Ex C_t$.

Note that the value function of an arbitrary policy $\pi$, i.e., $v^{\pi}(t,n)$, is the expected quality adjusted lifetime of a patient under that policy. In what follows, we let $v^{\pi}(t,n|\mathcal{E})$ represent the value function of the policy $\pi$ conditional on an event $\mathcal{E}$. For instance, $\big(v^{\pi_1}(t,n)- v^{\pi_2}(t,n) \big | C_t \le y \big)$ denotes the QALE difference between two arbitrary policies $\pi_1$ and $\pi_2$ conditional on the event  $C_t \le y$.
\begin{figure}[h!]
\centering
\includegraphics[scale=0.6]{./files/QALE-F1.pdf}
\caption{\boldmath Linking $v(t,n,y)$, $v(t+y,n,0)$, and $v^{\pi_0}(t+y,n)$ (recall that $v(t,0)=v^{\pi_0}(t,n)$ for all $t$).}
\label{fig:lem}
\end{figure}
\begin{lemma} \label{lem:delta}
The following equality holds for $v(t,n,y)$.
\begin{align*} 
v(t,n,y)&=\surv{C_t}{y}\bigg[ v(t+y,n,0)-v^{\pi_0}(t+y,n)\bigg]+v^{\pi_0}(t,n)
\end{align*}
\end{lemma}
\proof {Proof.}
Consider Figure \ref{fig:lem}. We want to prove that the difference between the value functions of the policy consisting of referral at $t+y$ for the current AVF chance and then the optimal policy for the subsequent decisions and the no-referral policy, i.e.,  $v(t,n,y) - v^{\pi_0}(t,n)$, equals $\surv{C_t}{y}\bigg[ v(t+y,n,0)-v^{\pi_0}(t+y,n)\bigg]$.
If $C_t \le y$, then the patient dies before the AVF referral, in which case there is no difference between the two policies. If $C_t > y$, on the other hand, then the difference between the two policies equals the difference between the value function at the state $(t+y,n)$ when we follow the policy consisting of immediate referral for the current AVF chance and then the optimal policy for the subsequent decisions and that of the same state but following the no-referral policy, i.e., $v(t+y,n,0)- v^{\pi_0}(t+y,n)$. Thus, we have:
\begin{align*} 
v(t,n,y)- v^{\pi_0}(t,n)=&\bigg[\big(v(t,n,y)- v^{\pi_0}(t,n)\big) \big | C_t \le y \bigg]\pr [C_t \le y]+\bigg[\big(v(t,n,y)- v^{\pi_0}(t,n)\big) \big | C_t > y \bigg]\pr [C_t > y]\\
&=\pr [C_t \le y](0)+\pr [C_t > y]\big[v(t+y,n,0)- v^{\pi_0}(t+y,n)\big]\\
&=\surv{C_t}{y}\big[ v(t+y,n,0)-v^{\pi_0}(t+y,n)\big].  \Halmos
\end{align*}

\endproof


For Lemma \ref{lem:dec_v}, let $w(t,m,k)$ denote the residual HD utility adjusted lifetime expectancy of a patient at time $t$ (which is the patient's QALE without subtracting the AVF creation disutility) under a scenario in which the patient undergoes the surgery at $t$ for her only AVF chance and the AVF maturation time and AVF lifetime are deterministically set at $m$ and $k$, respectively.

We can calculate $w(t,m,k)$ as follows:
\begin{align} \nonumber
w(t,m,k)=q_c\int_{0}^{m} x\pdf{C_t}{x}dx\\ \label{eq:wdef} 
 +\surv{C_t}{m}\bigg [q_cm+ & q_a\int_{0}^{k} x \pdf{A_{t+m}}{x} dx+\surv{A_{t+m}}{k} \big[q_ak+q_c\Ex C_{t+m+k} \big ] \bigg].
\end{align}

Since AVF creation variables $M$ and $K$ are assumed to be independent of a patient's survival by Assumption \ref{ass:AVFs}, we can express $v(t,1,0)$ and $v^{\pi_0}(t,n)$ using $w(t,m,k)$ as follows:
\begin{align}\label{eq:w_v1}
&v(t,1,0)=-d+\Ex_{M,K}[w(t,m,k)],\\ \label{eq:w_v0}
&v^{\pi_0}(t,n)=w(t,m,0):\forall m.
\end{align}
We will use these equalities in later proofs.
\begin{lemma} \label{lem:dec_v}
$ \diffp{w(t,m,k)}{k}$ is non-negative and decreasing in $t$ and $m$.
\end{lemma}

\proof {Proof.} To have differentiability of $w$ in $k$, it suffices to assume that $\surv{A}{x}$ and $\surv{C}{x}$ are differentiable at all values of $x$ because they in turn imply that $\surv{A_t}{x}$ and $\surv{C_t}{x}$ (as a direct result) and $\Ex C_{x}$ (using Lemma \ref{lem:res'}) are differentiable functions in $x$.

We have:
\begin{align}\nonumber
  \frac{\partial}{\partial k}w(t,m,k)
  =& \surv{C_t}{m} \bigg [ \diff{q_a\int_{0}^{k} x \pdf{A_{t+m}}{x} dx}{k} +\surv{A_{t+m}}{k} \diff{\big[q_ak+q_c\Ex C_{t+m+k} \big ]}{k}  \\ \label{eq:w'1}
 &+  \big [\diff{\surv{A_{t+m}}{k}}{k}\big ]\big[q_ak+q_c\Ex C_{t+m+k} \big ] \bigg ]\\ 
& =  \surv{C_t}{m} \bigg [ q_ak\pdf{A_{t+m}}{k}+\surv{A_{t+m}}{k} \big\{ q_a+q_c \big [\hrate{C_{t+m}}{k} \Ex C_{t+m+k}-1]\big\} \nonumber \\
&-\pdf{A_{t+m}}{k}  [q_ak+q_c \Ex C_{t+m+k} ]  \bigg] \label{eq:w'2} \\ 
&= \surv{C_t}{m} \surv{A_{t+m}}{k} \bigg[q_a-q_c+q_c \Ex C_{t+m+k} \big [\hrate{C_{t}}{m+k}-\hrate{A_t}{m+k}\big] \bigg]  \label{eq:w'}
\end{align}
where Equation \ref{eq:w'1} follows from Equation \ref{eq:wdef} using the product rule in calculating the derivatives of products of two functions, Equation \ref{eq:w'2} follows from Equation \ref{eq:w'2} by using Lemma \ref{lem:res'}, and  finally Equation \ref{eq:w'} follows from Equation \ref{eq:w'2} by rearranging terms.

We can prove that $\diffp {w(t,m,k)}{k}$ is decreasing in $t$ and non-negative by showing that it is a product of the following three non-negative decreasing functions:
\begin{enumerate}
\item $\surv{C_t}{m}$: This is decreasing in $t$, since $C_t$ is stochastically decreasing in $t$ based on Lemma \ref{lem:IFR} and that $C$ is IFR by Assumption \ref{ass:IFR}.
\item $\surv{A_{t+m}}{k}$: This is decreasing in $t$, since $A_{t+m}$ is stochastically decreasing in $t$ based on Lemma \ref{lem:IFR} and that $A$ is IFR by Assumption \ref{ass:IFR}.
\item $q_a-q_c+q_c \Ex C_{t+m+k} \big [\hrate{C_{t}}{m+k}-\hrate{A_t}{m+k}\big]$:
\begin{itemize}
\item non-negative: We have that $q_a \ge q_c$. Also, $\hrate{C_{t}}{m+k} \ge \hrate{A_t}{m+k}$ based on Lemma \ref{lem:assgen}.
\item decreasing: $\Ex C_{t+m+k}$ is decreasing in $t$, because $C_{t+m+k}$ is stochastically decreasing in $t$ by Lemma \ref{lem:IFR} and the fact that $C$ is IFR by Assumption \ref{ass:IFR}. Also, $\hrate{C_{t}}{m+k}-\hrate{A_t}{m+k}$ is decreasing in $t$ based on Lemma \ref{lem:assgen}.  
\end{itemize} 
Using the same logic, we can show that $\diffp {w(t,m,k)}{k}$ is decreasing in $m$.
\Halmos
\end{enumerate}
\endproof


\begin{lemma} \label{lem:v1}
$ v(t,1,0)-v^{\pi_0}(t,1)$ is decreasing in $t$.
\end{lemma}
\proof {Proof.} Choose $t_1 \le t_2$ arbitrarily. We have that $\forall m:\diffp {[w(t_2,m,k)-w(t_1,m,k)]}{k} \le 0$ by the linearity of the differential operator, and Lemma \ref{lem:dec_v}. This implies that $$\forall k,m: w(t_2,m,k)-w(t_1,m,k) \le w(t_2,m,0)-w(t_1,m,0).$$
But, $\forall m,t:w(t,m,0)=v^{\pi_0}(t,1) $. Thus, 
$$\forall k,m:  w(t_2,m,k)-w(t_1,m,k) \le v^{\pi_0}(t_2,1)-v^{\pi_0}(t_1,1).$$
Taking expectation from both sides with respect to $M,K$ gives us:
$$v(t_2,1,0)-v(t_1,1,0) \le v^{\pi_0}(t_2,1)-v^{\pi_0}(t_1,1). $$  By rearranging the terms in the above inequality we obtain the desired result. \Halmos
\endproof

\begin{proposition}[Existence of a Referral Threshold for $n=1$] \label{prop:qalen=1}
Assume $n=1$. Under Assumptions \ref{ass:dec}-\ref{ass:qol}, there is a threshold $\tau^*$ such that the policy $\pi(\tau^*)$ maximizes the QALE of the patient. In other words, for $t < \tau^*$, referral at $t$ is the optimal action, otherwise, the no-referral policy is optimal.
\end{proposition}

\proof {Proof.} Fix $t$, and $n$.  Assume that we take action $@_y$. By Lemma \ref{lem:delta}, we have:
\begin{align*}
v(t,1,y)=\surv{C_t}{y}\big[ v(t+y,1,0)-v^{\pi_0}(t+y,1)\big]+v^{\pi_0}(t,1).
\end{align*}
For $@_y$ to be an optimal action, it is necessary that referral at $t+y$ is no worse than the no-referral policy, i.e., $v(t+y,1,0) \ge v^{\pi_0}(t+y,1)$.\\
Since $v(t+y,1,0) - v^{\pi_0}(t+y,1)$ is decreasing in $y$ by Lemma \ref{lem:v1}, and $\surv{C_t}{y}$ is decreasing in $y$, then $v(t,1,y)$ is decreasing in $y$ for all $y$ that satisfy the necessary condition. Thus, the optimal action is to refer at $t$ if $v(t,1,0) \ge v^{\pi_0}(t,1)$, and no-referral, if otherwise.

Now, we form the policy $\pi(\tau^*)$ as follows based on whether $v(0,1,0) \le v^{\pi_0}(0,1)$ or not.
\begin{itemize}
\item $v(0,1,0) \le v^{\pi_0}(0,1)$: we have that for $\forall t:v(t,1,0) \le v^{\pi_0}(t,1)$, since $v(t,1,0) - v^{\pi_0}(t,1)$ is decreasing in $t$ by Lemma \ref{lem:v1}. As a result, the no-referral policy (i.e., ``CVC forever'') is optimal for all $t$. Choose  $\tau^*=0$ in this case.
\item $v(0,1,0) > v^{\pi_0}(0,1)$: we have that $\exists t'\le \infty$ such that for $t < t'$, we have $v(t,1,0) > v^{\pi_0}(t,1)$, and $v(t,1,0) \le v^{\pi_0}(t,1)$ for $t \ge t'$  because $v(t,1,0) - v^{\pi_0}(t,1)$ is decreasing in $t$. For $t <t'$, referral at $t$ is optimal, and for $t \ge t'$, the patient should remain on a CVC, i.e., the no-referral policy is optimal. Choose $\tau^*=t'$ in this case.
\end{itemize}
The policy $\pi(\tau^*)$ is optimal for $n=1$ by construction.\Halmos
\endproof


Now that we have established the optimality of threshold policies for $n=1$, we prove that this is the case for $n>1$, as well. To prove that, we restrict ourselves to threshold policies that are not dominated by $\pi_0$ when applied at any state $(t,n)$.
Let $T$ be the set of such thresholds, i.e., $$T=\{\tau \big| v^{\pi(\tau)}(t,n) \ge {v^{\pi_0}(t,n)} : \forall t,n  \}.$$ Note that $T$ is non-empty since $0 \in T$. The objective in Theorem \ref{thm:QALE} is to show that there exists $ \tau^* \in T$ such that $\pi(\tau^*)$ is optimal, and that $\tau^*$ is identical to the $\tau^*$ for the $n =1$ case constructed in Proposition \ref{prop:qalen=1}.

Proposition \ref{prop:thresh_dec} and Corollary \ref{cor:NWN} are used in proving Theorem \ref{thm:QALE}.

\begin{proposition} \label{prop:thresh_dec} For all $\tau \in T$, we have $\forall n: v^{\pi(\tau)}(t,n)- {v^{\pi_0}(t,n)}$ is decreasing in $t$.
\end{proposition} 
\proof {Proof.} 
We prove the result by induction on $n$ as follows:
\begin{itemize}
\item $n=1$: For $v^{\pi(\tau)}(t,1)$ we have:
\begin{align*}
v^{\pi(\tau)}(t,1)- v^{\pi_0}(t,1)=
\begin{cases}
v(t,1,0)-v^{\pi_0}(t,1)& t < \tau \\
0& o.w.
\end{cases}
\end{align*}

The function is decreasing  for $t \le \tau$ by Lemma \ref{lem:v1}, and for $t \ge \tau$ trivially. It suffices to have that $v^{\pi(\tau)}(\tau,1) - v^{\pi_0}(\tau,1) \ge 0$, which holds by the fact that $\tau \in T$.
\item Assume the result holds for $n=1,\ldots, k$. We prove that it holds for $n=k+1$.

Note that $\forall n:v^{\pi_0}(t,n)=v^{\pi_0}(t,0)$, as the no-referral policy never uses any AVF opportunities. We use $v^{\pi_0}(t,.)$ to denote the independence of the value of this policy on the remaining AVF chances $n$. 
To prove the result, it suffices to show the following:
\begin{align} \label{eq:relative}
\forall n>1: v^{\pi(\tau)}(t,n) - v^{\pi(\tau)}(t,n-1) \text{ is decreasing in } {t},
\end{align}
because then $v^{\pi(\tau)}(t,k+1)- {v^{\pi_0}(t,.)}=[v^{\pi(\tau)}(t,k+1)- v^{\pi(\tau)}(t,k)]+[v^{\pi(\tau)}(t,k)- {v^{\pi_0}(t,.)}]$ and the fact that the sum of two decreasing functions is decreasing will yield us the desired result.\\
Now we show Equation \ref{eq:relative} as follows. We first fix $M_i=m_i$ and $K_i=k_i$ for $i=1,\ldots,k$ arbitrarily. Since $M_i$, $K_i$ are independent of the survival process and the policy in use (based on Assumption \ref{ass:AVFs}), the result generalizes using the preservation of monotonicity under expectation.\\
Let $t':=\sum_{i=1}^{k} m_i+k_i$, and based on whether $ t+t' < \tau$, consider two cases:
\begin{enumerate}
\item [Case 1.] $t+t' < \tau$: In this case the last referral happens at time $t+t'$, provided that patient survives until that time. The difference in the cases of $k+1$, and $k$ remaining AVF chances is the (possible) use of one AVF chance. Let $S(t,t')$ represent the probability of survival of the patient until time $t'$. Then, we have the following:
\begin{align*}
v^{\pi(\tau)}(t,n)- v^{\pi(\tau)}(t,n-1)=S(t',t) \big[v^{\pi(\tau)}(t+t',1)-v^{\pi_0}(t+t',1)\big].
\end{align*}
\item [Case 2.] $t+t' \ge \tau$: We have that $\exists j \le k: t+\sum_{i=1}^{j} [m_i+k_i]\ge \tau$. Based on $\pi(\tau)$, we do not refer for the remaining AVF chances after $\tau$, and thus we have  
$v^{\pi(\tau)}(t,k+1)=v^{\pi(\tau)}(t,l)$ for all $j \le l$.
\end{enumerate}
Thus, we have
\begin{align}\label{eq:deltavn}
 v^{\pi(\tau)}(t,k+1)- v^{\pi(\tau)}(t,k)=
\begin{cases}
S(t',t) \big[v^{\pi(\tau)}(t+t',1)-v^{\pi_0}(t+t',1)\big]&:  t +t' <  \tau \\
0 &: o.w.
\end{cases}
\end{align}

It suffices to show that $S(t',t) \big[v^{\pi(\tau)}(t+t',1)-v^{\pi_0}(t+t',1)\big] $ is decreasing in $t$ and non-negative. We prove it by showing that it is the product of the following two non-negative and decreasing functions:
\begin{enumerate} 
\item $S(t',t)$: The probability is non-negative by definition. First we compute $S(t',t)$ as follows:
\begin{align} \nonumber
S(t',t)=&\pr [C_t>m_n , A_{t+m_n}>k_n,\ldots ,A_{t+t'-k_2}>k_2]\\ \label{eq:stprob1}
=&\pr[C_t>m_n] \pr[A_{t+m_n}>k_n|C_t>m_n]\ldots  \pr [A_{t+t'-k_2}>k_2|C_t>m_n ,\ldots]\\ \label{eq:stprob2}
=&\surv{C_t}{m_n}\surv{A_{t+m_n}}{k_n}\ldots \surv {A_{t+t'-k_2}}{k_2},
\end{align}
where Equation \ref{eq:stprob2} follows from Equation \ref{eq:stprob1} and Assumption \ref{ass:surv}. Each of the survival probabilities are decreasing in $t$ because $A_{t+x}$ and $C_{t+x}$ are stochastically decreasing in $t$, for any $x\ge 0$ by Lemmas \ref{lem:IFR} and \ref{lem:assgen}.
\item $v^{\pi(\tau)}(t+t',1)-v^{\pi_0}(t+t',1)$: This term is non-negative since $\tau \in T$. Also, this term is decreasing in $t$ using the result of the case $n=1$. \Halmos
\end{enumerate}
\end{itemize}
\endproof

\begin{corollary} \label{cor:NWN}
If $\forall t: v^{\pi(\tau)}(t,1) \ge v^{\pi_0}(t,1)$, then  $v^{\pi(\tau)}(t,n)$ is increasing in $n$ for all $t$. As a result  $\tau \in T$.
\end{corollary}
\proof {Proof.} 
Since $\forall t: v^{\pi(\tau)}(t,1) \ge v^{\pi_0}(t,1)$ by assumption, from Equation \ref{eq:deltavn}, we have that for any realization of $M_{1,\ldots,n},K_{1,\ldots,n}$:
 $$[ v^{\pi(\tau)}(t,n+1)- v^{\pi(\tau)}(t,n)]\big | M_{1,\ldots,n},K_{1,\ldots,n}  \ge 0.$$ Taking expectation with respect to $ M_{1,\ldots,n},K_{1,\ldots,n}$ and rearranging terms, we obtain:
\begin{align*} 
 \forall t , n: v^{\pi(\tau)}(t,n) \ge v^{\pi(\tau)}(t,n-1)\ge \ldots \ge v^{\pi(\tau)}(t,1) \ge v^{\pi_0}(t,1).  \Halmos
\end{align*}
\endproof
We now prove Theorem \ref{thm:QALE}.
\begin{repeattheorem} [Theorem \ref{thm:QALE}.] \text{\normalfont \textbf{(Optimality of Threshold Policies)}}
The policy $\pi(\tau^*)$ (constructed in Proposition \ref{prop:qalen=1}) is optimal for all $n \ge 1$.
\end{repeattheorem}

\proof {Proof.} We prove the theorem by induction on $n$:

\begin{itemize}
\item $n=1$: The policy $\pi(\tau^*)$ is optimal for $n=1$ by construction.
\item Assume for $n=1,\ldots, k$ the threshold policy $\pi(\tau^*)$ is optimal. We prove that, it is optimal for $n=k+1$ as well.\\
We prove the optimality of the policy $\pi(\tau^*)$ based on whether $t >\tau^*$ or not as follows.
\begin{enumerate}
\item [Case 1.] $t \ge \tau^*$: The policy suggests no more AVF-referral. We argue that it is optimal as follows.\\
Note that based on the optimality of $\pi(\tau^*)$ for $n \le k$, the last $k$ AVF chances won't be used, because their referral would happen at some $t' \ge t \ge \tau^*$.
Thus, we are left with one AVF chance. Similarly, we should not use that chance either. Thus, the no-referral policy is optimal in this case.

\item [Case 2.] $t < \tau^*$: The policy suggests referral at $t$. We argue that it is optimal as follows.\\
Note that no referral can be made later than $\tau^*$ (using the logic explained in the first case). Thus, we restrict our attention to $y < \tau^*-t$. For all such $y$, we have that $v(t+y,n,0)= v^{\pi(\tau ^*)}(t+y,n)$, based on the induction assumption. By Lemma \ref{lem:delta}, we have
$$v(t,n,y)=\surv{C_t}{y}\bigg[v^{\pi(\tau ^*)}(t+y,n)-v^{\pi_0}(t+y,n)\bigg]+v^{\pi_0}(t,n).$$ 


 We conclude the proof by showing $v(t,n,y)$ is decreasing in $y$. Since $\surv{C_t}{y}$ is decreasing in $y$, it suffices to show that $v^{\pi(\tau ^*)}(t+y,n)-v^{\pi_0}(t+y,n)$ is non-negative and decreasing in $y$. Since $\pi(\tau^*)$ is optimal for $n=1$, then $\forall t: v^{\pi(\tau^*)}(t,1) \ge v^{\pi_0}(t,1)$. As a result of Corollary \ref{cor:NWN}, we have $\tau^* \in T$. This implies  $v^{\pi(\tau ^*)}(t+y,n) \ge v^{\pi_0}(t+y,n).$ Since $\tau^* \in T$, using Proposition \ref{prop:thresh_dec}, we have that $v^{\pi(\tau ^*)}(t+y,n)- v^{\pi_0}(t+y,n) \text{ is decreasing in } y$. \Halmos
\end{enumerate}
\end{itemize} 
\endproof
\subsection{Proof of Proposition \ref{prop:binsearch}} 
\begin{repeattheorem}[Proposition \ref{prop:binsearch}.] \text{\normalfont \textbf{(Binary Search)}}
An optimal threshold policy can be found using a binary search for $\tau^*$ over $[0,t_{\max}]$, where $t_{\max}$ is a reasonable upper bound for $\tau^*$.
\end{repeattheorem}

\proof {Proof.} Based on the way the optimal policy is formed in Proposition \ref{prop:qalen=1}, we have that for all $t \in (0, \tau^*)$, $v(t,1,0) > v^{\pi_0}(t,1)$ and for all $t \in [\tau^*, t_{\max}]$, we have $v(t,1,0) \le v^{\pi_0}(t,1)$. Since $v(t,1,0) - v^{\pi_0}(t,1)$  is a decreasing continuous function, we can find $\tau^*$ using a binary search over $[0,t_{\max}]$.
\Halmos
\endproof

\subsection{Proof of Proposition \ref{prop:dec_d}} 
\begin{repeattheorem} [Proposition \ref{prop:dec_d}.]
The critical HD duration is decreasing in AVF creation disutility, $d$.
\end{repeattheorem}
\proof{Proof.}
Choose $0 \le d_1 \le d_2$ arbitrarily. By Theorem \ref{thm:QALE}, there exist $\tau^*(d_1)$ and $\tau^*(d_2)$ for which threshold policies $\pi(\tau^*(d_1))$ and $\pi(\tau^*(d_2))$ are optimal for cases $d=d_1$  and $d=d_2$, respectively.
Based on the way we constructed $\tau^*(d)$ in Proposition \ref{prop:qalen=1}, we have:
\begin{align} \label{eq:IFFTHLD}
t \ge \tau^*(d) \iff v(t,1,0;d ) \le  v^{\pi_0}(t,1)
\end{align}
By Equation \ref{eq:IFFTHLD}, we have $v(\tau^*(d_1),1,0 ;d_1) \le v^{\pi_0}(\tau^*(d_1),1)$ (substitute $\tau^*(d_1)$ for $t$ and $d_1$ for $d$). We have $\forall s: v(s,1,0 ;d) - v^{\pi_0}(s,1)$ is decreasing in $d$ (see Equation \ref{eq:w_v1}). Thus, $v(\tau^*(d_1),1,0 ;d_2) \le v(\tau^*(d_1),1,0 ;d_1) \le v^{\pi_0}(\tau^*(d_1),1)$. Therefore, by Equation \ref{eq:IFFTHLD}, we have $ \tau^*(d_2) \le \tau^*(d_1)$ (substitute $\tau^*(d_1)$ for $t$ and $d_2$ for d). \Halmos
\endproof

\subsection{Proof of Theorem \ref{thm:cdis} and Corollary \ref{cor:dcrmap}}
\begin{repeattheorem}[Theorem \ref{thm:cdis}.] \text{\normalfont \textbf{(Critical Disutility)}} For any HD duration $t$, there exists a non-negative critical AVF creation disutility, denoted by $d^{\text{cr}}(t)$, such that the optimal decision at time $t$ is to do an AVF surgery immediately if the patient's AVF creation disutility is less than the critical disutility (i.e., if $d < d^{\text{cr}}(t)$), and is to use CVC for the rest of patient's life, otherwise.
\end{repeattheorem}

\proof {Proof.} Fix $t$. Define $d^{\text{cr}}(t)$ as follows:
\begin{align} \label{eq:cdisdef}
d^{\text{cr}}(t):=\Ex_{M,K}[w(t,m,k)]- v^{\pi_0}(t,1)
\end{align}
and note that $d^{\text{cr}}(t)=d+v(t,1,0) -v^{\pi_0}(t,1)$ (see Equation \ref{eq:w_v1}).

By Equation \ref{eq:IFFTHLD}, we have that $t < \tau^*(d)$, and immediate surgery is optimal, if $d^{\text{cr}}(t) > d$ and using a CVC for the rest of the patient's lifetime is optimal, otherwise.  Thus, $d^{\text{cr}}(t)$ is indeed a critical value for AVF creation disutility in determining the optimal policy. \Halmos
\endproof
Note that we can use Equation \ref{eq:cdisdef} to numerically calculate the critical disutility by calculating
$\Ex_{M,K}[w(t,m,k)]$, either by Monte-Carlo simulation or analytically, and $v^{\pi_0}(t,1)$ using the equality $v^{\pi_0}(t,1)=q_c\Ex C_t$.

We use the following lemma to prove Corollary \ref{cor:dcrmap}.
\begin{lemma} \label{lem:dwdmdk}  $w(t,m,k)$ is increasing in $k$ and decreasing in $m$.
\end{lemma}
\proof{Proof.} 
By Lemma \ref{lem:v1}, $\diffp{w(t,m,k)}{k} \ge 0$, and thus $w(t,m,k)$ is increasing in $k$. Using the same lemma, we have that $\diffp{w(t,m,k)}{k}$ is decreasing in $m$. Choose $m_1 \le m_2$ arbitrarily. We have that $\forall t:\diffp{[w(t,m_2,k)-w(t,m_1,k)]}{k} \le 0$ by the linearity of the differential operator. This implies that 
$$\forall k,t: w(t,m_2,k)-w(t,m_1,k) \le w(t,m_1,0)-w(t,m_2,0)=0.$$
and thus the result. The last equality is based on the fact that $\forall t,m: w(t,m,0)=v^{\pi_0}(t,1) $. \Halmos
\endproof


\begin{repeattheorem} [Corollary \ref{cor:dcrmap} \text{\normalfont \textbf{(Relationship between Critical Disutility and Critical Duration)}}.]
Let $\tau^*(d)$ be an optimal HD duration threshold for a patient with AVF creation disutility $d$. Then, if  Assumptions \ref{ass:IFR} and \ref{ass:qol} hold strictly, we have that $d^{cr}(t)$ is strictly decreasing and thus invertible. If in addition we assume that $\lim\limits_{t \to \infty} \hrate{A}{t} =\infty$, then we have
\begin{align*} %\label{eq:equiv}
\tau^*(d)=\begin{cases}
d^{-1}_{cr}(d) &:  0 < d < d^{cr}(0)\\
0&:  d \ge d^{cr}(0)
\end{cases}
\end{align*}
where $d^{-1}_{cr}(d)$ represent the inverse function of $d^{cr}(t)$.
\end{repeattheorem}

\proof{Proof.}
Since Assumptions \ref{ass:IFR} and \ref{ass:qol} hold strictly, we have that the monotonicity is strict for Lemmas \ref{lem:dec_v} and \ref{lem:v1}. Therefore, we have that $d^{\text{cr}}(t)$ is strictly decreasing since $d^{\text{cr}}(t)=d+v(t,1,0) -v^{\pi_0}(t,1)$. We next prove that $\lim_{t \to \infty} d^{\text{cr}}(t)=0 $ and thus $d^{\text{cr}}(t)$ is invertible over $(0,d^{\text{cr}}(0)]$.

Recall that $d^{\text{cr}}(t)=\Ex_{M,K}[w(t,m,k)]- q_c\Ex C_t$. Since $w(t,m,k)$ is decreasing in $m$ and increasing in $k$ by Lemma \ref{lem:dwdmdk}, we have that $$\Ex_{M,K}[w(t,m,k)] \le w(t,0,\infty) =q_a\Ex A_t,$$
where the equality follows from the fact that $w(t,0,\infty)$ is  the residual utility adjusted lifetime expectancy of a patient at time $t$ when the patient is on AVF from $t$ until death (see the definition of $w$ in Equation \ref{eq:wdef}). As a result, $\forall t:d^{\text{cr}}(t) \le [q_a\Ex A_t - q_c \Ex C_t]$.

Let $X(t)$ be an exponentially distributed random variable with hazard rate $\hrate{A}{t}$. Then, we have $A_t \le_{hr} X(t)$ and as a result $\Ex A_t \le \Ex X(t) = \frac{1}{\hrate{A}{t}}$. Therefore, $\lim_{t \to \infty} \Ex A_t =0$ because $\lim\limits_{t \to \infty} \hrate{A}{t} =\infty$. Since $\forall t:C_t \le_{st} A_t$, we have that $\forall t: \Ex C_t \le \Ex A_t$. Based on the fact that $\lim_{t \to \infty} \Ex A_t =0$, we obtain $\lim_{t \to \infty} \Ex C_t =0$ as well. Using the fact that $\forall t:0 \le d^{\text{cr}}(t) \le [q_a\Ex A_t - q_c \Ex C_t]$ and that  $\lim_{t \to \infty} \Ex C_t =\lim_{t \to \infty} \Ex A_t=0$, we obtain $\lim_{t \to \infty} d^{\text{cr}}(t)=0 $.

Fix $d$ arbitrarily such that $0 < d < d^{cr}(0)$. For all $ t < d^{-1}_{cr}(d)$, we have $d < d^{cr}(t)$ and thus by Theorem \ref{thm:cdis}, the optimal decision at time $t$ is to do an AVF surgery immediately and for all $t \ge d^{-1}_{cr}(d)$, we have that $d \ge  d^{cr}(t)$ and thus by the same theorem, the optimal decision at time $t$ is to use CVC for the rest of the patient's life. Therefore, $d^{-1}_{cr}(d)$ is indeed a critical HD duration for a patient with AVF creation disutility $d$.

For $d \ge d^{cr}(0)$, we have that $v(0,1,0) \le v^{\pi_0}(0,1)$ and thus by the way we constructed $\tau^*(d)$ in Proposition \ref{prop:qalen=1}, we have that $\tau^*(d)=0$. \Halmos
\endproof


\subsection{Proof of Theorem \ref{thm:compdcrt}}
Before proving Theorem \ref{thm:compdcrt}, we prove Proposition \ref{prop:compdcr}, which is used in comparing the critical disutility under different scenarios.

\begin{proposition}\label{prop:compdcr} Consider two patients types indexed by 1 and 2 whose characteristics are such that all of following hold:
\begin{enumerate}
\item $M^{(2)} \le_{st} M^{(1)}$
\item $K^{(1)} \le_{st} K^{(2)}$
\item $\forall m,k, s\ge t: \diffp{w_{(1)}(s,m,k)}{k} \le \diffp{w_{(2)}(s,m,k)}{k}$.
\end{enumerate}
Then, $d^{\text{cr}}_{(1)}(t) \le d^{\text{cr}}_{(2)}(t)$.
\end{proposition}

\proof{Proof}
 Let  $d^{cr}(t,m, k):=w(t,m,k)-v^{\pi_0}(t,1)$. Note that $d^{cr}(t)=\Ex_{M,K}[d^{cr}(t,M, K)]$. 
Based on assumption 3 of the proposition and the linearity of the differential operator, we have:
\begin{align*} 
\forall t,m,k:  w_{(1)}(t,m,k)-w_{(2)}(t,m,k) \le w_{(1)}(t,m,0)-w_{(2)}(t,m,0) =v_{(1)}^{\pi_0}(t,1)-v_{(2)}^{\pi_0}(t,1),
\end{align*}
and hence by rearranging terms we obtain
\begin{align} \label{eq:lempcomp}
\forall t,m,k; d^{cr}_{(1)}(t,m,k) \le d^{cr}_{(2)}(t,m, k).
\end{align}
As a corollary to Lemma \ref{lem:dwdmdk}, we have that $d^{cr}(t,m, k)$ is increasing in $k$ and decreasing in $m$. Therefore by Lemma \ref{lem:fincstorder}, we have:
\begin{align*}
M^{(2)} \le_{st} M^{(1)} \text{ and } K^{(1)} \le_{st} K^{(2)} \implies d^{cr}_{(1)}(t,M_1, K_1) \le_{st} d^{cr}_{(1)}(t,M_2, K_2).
\end{align*}
Taking expectation from both sides, we get:
\begin{align*}
d^{cr}_{(1)}(t)=\Ex_{M_1,K_1} [d^{cr}_{(1)}(t,M_1, K_1)] \le \Ex_{M_2,K_2} [d^{cr}_{(1)}(t,M_2, K_2)].
\end{align*}
Also, taking expectation from both sides of inequality \ref{eq:lempcomp}, we get:
\begin{align*}
\Ex_{M_2,K_2} [d^{cr}_{(1)}(t,M_2, K_2)] \le \Ex_{M_2,K_2} [d^{cr}_{(2)}(t,M_2,K_2)]=d^{cr}_{(2)}(t).
\end{align*}
Combining the last two inequalities, we get the desired result.
\Halmos
\endproof


\begin{repeattheorem} [Theorem \ref{thm:compdcrt}.]
Consider two patients types indexed by 1 and 2 whose characteristics satisfy all of the following properties:
\begin{enumerate}
\item $M^{(2)} \le_{st} M^{(1)}$
\item $K^{(1)} \le_{st} K^{(2)}$
\item $q_c^{(1)} \le q_c^{(2)}$
\item  $q_a^{(1)} - q_c^{(1)} \le q_a^{(2) }- q_c^{(2)}$
\item $C^{(1)} \le_{hr} C^{(2)}$
\item $[\hrate{C^{(1)}}{t}- \hrate{A^{(1)}}{t}] \le [\hrate{C^{(2)}}{t} -\hrate{A^{(2)}}{t}]: \forall t$
\end{enumerate}
where $(i)$ denotes the patient's index. Then, $d^{\text{cr}}_{(1)}(t) \le d^{\text{cr}}_{(2)}(t): \forall t$.
\end{repeattheorem}
\proof{Proof.} Using Proposition \ref{prop:compdcr}, it suffices to show that $\forall m,k, s: \diffp{w_{(1)}(s,m,k)}{k} \le \diffp{w_{(2)}(s,m,k)}{k}$. Recall from Equation \ref{eq:w'}, we have
\begin{align*}
\diffp {w(t,m,k)}{k}=\surv{C_t}{m} \surv{A_{t+m}}{k} \bigg[q_a-q_c+q_c \Ex C_{t+m+k} \big [\hrate{C_{t}}{m+k}-\hrate{A_t}{m+k}\big] \bigg]
\end{align*}

Assumption 5 of the theorem and Lemma \ref{lem:IFR} imply that $\surv{C_t}{m}^{(1)} \le \surv{C_t}{m}^{(2)} $ and $ \Ex C_{t+m+k}^{(1)} \le  \Ex C_{t+m+k}^{(2)}$. By combining assumptions 5 and 6 in this theorem, we have that $A^{(1)} \le_{hr} A^{(2)}$, which in turn implies $\surv{A_{t+m}}{k}^{(1)} \le \surv{A_{t+m}}{k}^{(2)} $. Combining these results with assumptions 3, 4 and 6, we obtain the desired result. \Halmos
\endproof

\subsection{Proof of Theorem \ref{thm:pavf}}
\begin{repeattheorem}[Theorem \ref{thm:pavf}.]
Critical disutility is proportional to the AVF creation success probability.
\end{repeattheorem}
\proof{Proof.}
Recall from Equation \ref{eq:cdisdef} that $d^{\text{cr}}(t)=\Ex_{M,K}[w(t,m,k)]- v^{\pi_0}(t,1)$. Let $Z$ be a random variable denoting the AVF lifetime conditional on the AVF creation being successful, i.e., when $K>0$. Then, we have
\begin{align} \label{eq:dcrprop0}
d^{\text{cr}}(t)
&=\pr [K=0]\Ex_{M}[w(t,m,0)]+\pr [K>0]\Ex_{M,Z}[w(t,m,z)]-v^{\pi_0}(t,1)\\ \label{eq:dcrprop1}
&=\pr [K=0]v^{\pi_0}(t,1)+\pr [K>0]\Ex_{M,Z}[w(t,m,z)]-v^{\pi_0}(t,1)\\
&=\pr [K>0]\big(\Ex_{M,Z}[w(t,m,z)] -v^{\pi_0}(t,1)\big)\label{eq:dcrprop2},
\end{align}
where Equation \ref{eq:dcrprop1} follows from Equation \ref{eq:dcrprop0} by using the fact $v^{\pi_0}(t,1)=w(t,m,0)$ (see Equation \ref{eq:w_v0}), and Equation \ref{eq:dcrprop2} follows from Equation \ref{eq:dcrprop1} by rearranging terms  and using the fact $\pr [K=0]=1-\pr [K>0]$.\Halmos
\endproof

\end{APPENDICES}


%------------------------------------------

% Appendix here
% Options are (1) APPENDIX (with or without general title) or
%             (2) APPENDICES (if it has more than one unrelated sections)
% Outcomment the appropriate case if necessary
%
% \begin{APPENDIX}{<Title of the Appendix>}
% \end{APPENDIX}
%
%   or
%
% \begin{APPENDICES}
% \section{<Title of Section A>}
% \section{<Title of Section B>}
% etc
% \end{APPENDICES}

%%
%\theendnotes

% Acknowledgments here
\ACKNOWLEDGMENT{
Reza Skandari is supported by Discovery Grant 341415-07 from the Natural Sciences and Engineering Research Council. Steven Shechter is supported by the Career Investigator Award of the Michael Smith Foundation for Health Research. 
}


% References here (outcomment the appropriate case)

% CASE 1: BiBTeX used to constantly update the references
%   (while the paper is being written).
%\bibliographystyle{ormsv080} % outcomment this and next line in Case 1
%\bibliography{<your bib file(s)>} % if more than one, comma separated

% CASE 2: BiBTeX used to generate mypaper.bbl (to be further fine tuned)
%\input{mypaper.bbl} % outcomment this line in Case 2

%If you don't use BiBTex, you can manually itemize references as shown below.

\bibliographystyle{ormsv080}
\bibliography{refs}

%%%%%%%%%%%%%%%%%
\end{document}
%%%%%%%%%%%%%%%%%

LEFTOVERS:
 in a systematic way, yet leave us with some freedom and choice
 
 The focus of this research was on the vascular access choice for patients who are already on HD. \cite{AVFpaper} investigated different AVF referral polices for maximizing a CKD patient's expected lifetime and used the policy in Section \ref{sec:optTL} (whose optimality was proven in this paper) for the HD period. As a future research, the optimal AVF referral policy maximizing a CKD patient's QALE can be studied. Obviously, the optimal policy described in Section \ref{sec:QALEGP} can be used for the period of time a kidney disease patient is on HD.
 
 -----------------------------------

% % % % % % % %
To do so, however, requires that we address how to deal with censored patient survival curves.  This is a modeling issue that repeatedly comes up in medical decision making problems. We address this practical modeling issue both analytically and numerically.
% % % % % % % % % % % %
Note that since $M_i,K_i$ are independent of the survival process and the policy in use (based on Assumption \ref{ass:AVFs}), we have that 
\begin{align*}
v^{\pi}(t,n)=\Ex \big [ v^{\pi}(t,n \big |M_1,\ldots, M_n,K_1,\ldots, K_n)\big].
\end{align*}
